:W&Bá¾ÎÜ/data/spatop/options_files/sweep_fixed_dp/config_g2vc1w2g.json*-l* /scratch/logs*-n*dpfx_vanilla_g2vc1w2g2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb%dp-spatop-sweepagent-vanilla-v2-h9cgsr/opt/conda/bin/pythonº mns3l5e5or9vut5nxqff9hd4z0l28ofqîÊÖyÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€€Âõª€àçÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-0c583aac-6f68-6140-7078-f37971b4b9bbÊ13.2º mns3l5e5or9vut5nxqff9hd4z0l28ofqµª½Ç2  requirements.txtÛÇ°àW g2vc1w2gZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64Kîý¹Y g2vc1w2gZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64ƒm’*  g2vc1w2gb trainer/global_step2­BbH\  g2vc1w2gZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64 ‹å/  g2vc1w2gb*"trainer/global_step2 ­Î­K  g2vc1w2gj: §¯çÒØ•ýÎ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] =˜fé’  g2vc1w2gj€  ¨¯çÒÈ£ú2î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 19.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ "l!q[ g2vc1w2gjÉ ¨¯çÒÐÙò3·Trainable params: 21.2 M Non-trainable params: 50 Total params: 21.2 M Total estimated model params size (MB): 84.814 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 ëpflp g2vc1w2gj_ ¨¯çÒ ¡Š4Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). /åz g2vc1w2gji ¨¯çÒØ—•4Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). z´½âp g2vc1w2gj_ ¨¯çÒ¸Úž4Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). bjý[s g2vc1w2gjb ¨¯çÒ¨ñ¦4Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ßýžÚm g2vc1w2gj\ ¨¯çÒ˜¨Î4Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). i~Rð g2vc1w2g*Þ  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true # use_pairwise_interactions‚false  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"ÀÎê g2vc1w2gjØ ¨¯çÒøòõ7È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. m±ÄM g2vc1w2gj» ¨¯çÒè÷ƒ8«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. |ÿ*3ë g2vc1w2gjÙ ª¯çÒ€ÏÕÁÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. mLóEP g2vc1w2gj¾ ª¯çÒè¦åÁ­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Gѵdž‚(:” ¶¯çÒ°¶¤ gpu.0.gpu‚89.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JUèé‚(:ß ¶¯çÒâ¡Ü¬) disk./.usagePercent‚36.77540933147053$ disk./.usageGB‚547.9801750183105 disk.rbd2.out‚0 disk.nvme0n1.out‚ 7.93359375 network.recv‚39040 disk.sda3.in‚0 disk.rbd1.out‚0.0625 disk.drbd2165.out‚0! disk.nvme0c0n1.out‚ 7.93359375 disk.sda2.in‚0$ memory_percent‚96.93911075592041( proc.memory.availableMB‚ 501.49609375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 7.93359375 disk.dm-1.out‚ 0.21484375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 cpu‚16.830697152205648 disk.sda3.out‚ 0.21484375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 0.21484375 disk.dm-0.in‚ 5.77734375 network.sent‚32954 disk.nvme0c0n1.in‚ 5.77734375 disk.sda2.out‚0 proc.memory.rssMB‚2635.75) proc.memory.percent‚16.08734130859375 disk.sda1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 5.77734375 disk.drbd2165.in‚0 disk.rbd0.in‚0Ѓ¿t‘‚(:‡ ůçÒÄé¤ gpu.0.gpu‚76.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&m…Lø‚(:î ůçҧ͟§ network.recv‚42853 disk.rbd2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 0.39453125' proc.memory.availableMB‚ 500.7421875 disk.sda.in‚0) disk./.usagePercent‚36.77541368359861 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0$ disk./.usageGB‚547.9802398681641 disk.dm-0.out‚19.00537109375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 0.39453125 cpu‚18.66044219002093 disk.rbd1.out‚0.0625 disk.nvme0c0n1.in‚ 6.60546875 disk.sda.out‚ 0.39453125 disk.sda3.in‚0$ memory_percent‚96.94371223449707 network.sent‚43564 disk.rbd1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2635.76953125 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚19.00537109375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 6.60546875# disk.nvme0n1.out‚19.00537109375 disk.drbd2165.in‚0 disk.dm-0.in‚ 6.60546875( proc.memory.percent‚16.0874605178833 proc.cpu.threads‚17+vjÃ3b/ lr-AdamW/pg1"trainer/global_step2:JP#©Îù3b/ lr-AdamW/pg2"trainer/global_step2:JPK=ø…-) _wandbruntime‚43  _runtime‚43 ?±( ½ ' lr-AdamW/pg1‚5.7901856206031056e-05 ' lr-AdamW/pg2‚5.7901856206031056e-05 trainer/global_step‚49 % _timestamp‚1.7842728518238676e+09 _runtime‚ 43.950550056 _step‚0©+4„Ä!¿ % _timestamp‚1.7842728518238676e+09 _runtime‚ 43.950550056 _step‚0 ' lr-AdamW/pg1‚5.7901856206031056e-05 ' lr-AdamW/pg2‚5.7901856206031056e-05 trainer/global_step‚49 {o÷j"bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP_R††m#bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPÙ‹±‰_$b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPúÎl%bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPÛ~Ì k&bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPŽ›Êg'bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP~€ÑŒo(bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPYP^)bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPÞ+Ë{n*bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP§=Ÿ2m+bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP‡:ø1l,bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP—VöPl-bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP™?†s.bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP!Šc/b_ ¢Kg:bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPú·äq;bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP[]}–sba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPÆ6¯/p?bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPuÁ€Âh@bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPÕ•gAbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP þCkBbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP{pŽ&rCbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPbœwiDbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP„ ±hEbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPiE ÖpFbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPD¥±ÞlGbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP[‹ÎarHbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP‡6BeIba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPV-ÏiJbe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPæxyiKbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPÏÈboLbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP VcMb_ ÞImObi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPTüUgPbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP_™ú5aQb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP/œÁöhRbd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP?~T‰lSbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPiû·égTbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPÀük_dUb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP¦£¾fVbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPº¦Š«oWbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP¤–âëhXbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPÏcqYbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP ;c`Zb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP‘hxi[be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPÿJÖr\bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPŠXp]bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPŒMÖn^bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPûYŽ__b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP¯ß³k`bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPáÂpabl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPŒìžcnbbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPTõz\kcbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPŸä^xdbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP _Íhebd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPV›;’pfbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPÚKxnngbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPÍÓvhbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP…~ˆ†sibo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP­)H{fjbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPŒëjkbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPèhØÈflbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPýÉlmbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPÔÞÝpnbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPyänobj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPFÁómpbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP#¦tqbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP ›Ýarb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPʪÌÒisbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP‡9;ltbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP¾,Psubo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP¾[ÊE`vb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP$«"Ð_wb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPhG/'xxbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPKI3iybe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP’~nzbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPÁ§j{bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP}Àxƒd|b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP áÙb}b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPÛ[6êk~bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP*!$âfbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPiÔf€ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPb9eXbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP?|èf‚ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPk–¬nƒbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPÏæP h„bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPN¸of…ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPôl•g†bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP\Yãh‡bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPÀøÐsˆbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP§ï5È^‰bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPtâN%oŠbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPGr‹bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPc®hŒbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP[Cùñkbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPšKjkŽbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPÚ4¢Ùjbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP“Y5úlbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPZ‰‡žo‘bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPÈ.–Šj’be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPCoW%g“bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP~jÉøt”bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPü—)êp•bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPL„:Ñn–bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP’¬]ñw—br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPqܘÄi˜bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP-ú)¼t™bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP$.Jkšbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPR^"ýo›bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPÌ`K±dœb_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11748 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚385460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 UKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93131 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚451528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554577408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554577408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 51547677030 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 51207128668 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74076672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 51627787264 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385460 % _timestamp‚1.7842728518246799e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚846616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 51627787264 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7412365087078 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7449162053120 LBDeviceStatsMonitor.on_train_b›\ÃÌ atch_start/allocation.small_pool.peak‚11748 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7360800370688 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚385460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 UKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93131 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚451528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554577408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554577408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 51547677030 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 51207128668 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74076672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 51627787264 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385460 % _timestamp‚1.7842728518246799e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚846616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 51627787264 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7412365087078 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7449162053120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11748 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7360800370688 GDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11748 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Ô»ÚPl bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPBMrw¡br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP¼¶Ûêk¢bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPÛl·žg£bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPÎ8š}w¤br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPcuªÙk¥bf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPZÓk¦bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP?âSs§bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPÃsme¨b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP~Œ£q©bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP{´V"dªb_ ç¤ûi«bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPy ýNm¬bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP9̘Íd­b_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP•EÍØlÄbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPe™ÀøfÅba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPWs!GbÆb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPwܺkÇbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPÕ´4mmÈbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPr©¬lÉbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPBÉg—uÊbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP5k(fËba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPš´ÇjÌbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPtôÄ–mÍbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPmécÎb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPZÕÇ"aÏb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JP·ÑÝjãbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPسiäbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPLÄü¨jåbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP›ólgæbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPê›OXmçbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP€dÖheèb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP·É›]ébX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP@ˆ9nêbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPt`iNaëb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPÈ€ïKiìbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP.#ÀIjíbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP;U rîbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÚÿPjïbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPš…Q­fðba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP‡$Oœoñbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP à¥ðpòbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP¬ÝmZ^óbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP]‰o@eôb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP »à dõb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP/îRp÷bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP±U‚Ïhøbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP±F~‰jùbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPÏéO×múbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP½E¨Ckûbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP£Þ’hübc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPÔ7­Ñ]ýbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPK(½òpþbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPÖ;¼­hÿbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPó:pk€bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPDæ|Wqbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPই:_‚bZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP›yìxeƒb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPÁÍ9e„b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPûÍ=i…bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPwªÙfu†bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPÿâ´h‡bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPY3mˆbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP#ô~o‰bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP_Ò±¨iŠbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPÁ\ r‹bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP 5zfŒba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPÓ Ÿ€pbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPGõ¸·VŽbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPì•ÎYmbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPæfËfba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPKvÙi‘bd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP$ò¯že’b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP¸Á*ån“bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPRR€§\”bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPQcãžg•bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP®Eü_n–bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPøòççq—bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÄQí`˜b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPÄßy¾f™ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPÅ]Álšbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPL±–.›) _wandbruntime‚44  _runtime‚44Sõ_øœ‹M Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7559529224950 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7545116454912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚393092 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚206065 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚96795136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7545133494272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚460580 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7545116454912 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚111189 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393092 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚470220 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 52640421376 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚205808 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1882542636544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚513 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 101070336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7507311042560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1882441566208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1828845561856 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 52299254784 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7597415709696 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 53697074688 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7559886812784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1828826949632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 F8DeviceStatsMonito³Þƒ3‚ r.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 44.567564828 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7597773915648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 53614616576 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 52218182390 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚863314 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7597773915648 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚110935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚448 trainer/global_step‚49 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842728524411793e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 52640421376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 52299254784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7507328081920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚863314 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7597773915648 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚110935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚448 trainer/global_step‚49 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842728524411793e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 52640421376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 52299254784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7507328081920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7545133494272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚460580 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7545116454912 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚111189 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393092 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚470220 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 52640421376 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚205808 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1882542636544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚513 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 101070336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7507311042560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1882441566208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1828845561856 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 52299254784 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7597415709696 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 53697074688 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7559886812784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1828826949632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 44.567564828 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7597773915648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 53614616576s‰ÏËCžb> loss/SRqqt2/assignment_loss"trainer/global_step2:JP•ˆ'ìAŸb< loss/FBt2/assignment_loss"trainer/global_step2:JP¢({[B b= loss/SRqqt1/detection_loss"trainer/global_step2:JP°”qb@¡b; loss/FBt2/detection_loss"trainer/global_step2:JP @GÕ-¢b( epoch"trainer/global_step2:JP¤vA£b< loss/FRt1/assignment_loss"trainer/global_step2:JP¾Ñ%ãC¤b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPäšP¥A¥b< loss/FBt1/assignment_loss"trainer/global_step2:JP‘ñ@@¦b; loss/FBt1/detection_loss"trainer/global_step2:JPH4A§b< loss/FRt2/assignment_loss"trainer/global_step2:JPÀÔ§…@¨b; loss/FRt1/detection_loss"trainer/global_step2:JPhÉŸ@©b; loss/FRt2/detection_loss"trainer/global_step2:JPí~3Bªb= loss/SRqqt2/detection_loss"trainer/global_step2:JPžúî7«b2 loss/total_loss"trainer/global_step2:JPø ä.¬) _wandbruntime‚44  _runtime‚44N²œˆõ­ï +loss/FBt2/detection_loss‚8.333504495911 epoch‚0 $ _timestamp‚1.784272852443563e+09 /loss/FRt1/assignment_loss‚4.508231659100447 2loss/SRqqt1/assignment_loss‚4.2544916242475255 /loss/FBt1/assignment_loss‚1.884366277191374 .loss/FBt1/detection_loss‚40.62677484750748 _step‚3 /loss/FRt2/assignment_loss‚5.337391606870129 /loss/FRt1/detection_loss‚0.3989612209714308 /loss/FRt2/detection_loss‚1.5945512848263972 1loss/SRqqt2/detection_loss‚1.4697960446566103 %loss/total_loss‚79.55896035631372 trainer/global_step‚49 _runtime‚ 44.569833802 2loss/SRqqt2/assignment_loss‚4.1903165810191725 /loss/FBt2/assignment_loss‚1.797070229613541 1loss/SRqqt1/detection_loss‚5.1635044843986115ßÈÄù®ó 2loss/SRqqt2/assignment_loss‚4.1903165810191725 /loss/FBt2/assignment_loss‚1.797070229613541 1loss/SRqqt1/detection_loss‚5.1635044843986115 +loss/FBt2/detection_loss‚8.333504495911 epoch‚0 $ _timestamp‚1.784272852443563e+09 /loss/FRt1/assignment_loss‚4.508231659100447 2loss/SRqqt1/assignment_loss‚4.2544916242475255 /loss/FBt1/assignment_loss‚1.884366277191374 .loss/FBt1/detection_loss‚40.62677484750748 _step‚3 /loss/FRt2/assignment_loss‚5.337391606870129 /loss/FRt1/detection_loss‚0.3989612209714308 /loss/FRt2/detection_loss‚1.5945512848263972 1loss/SRqqt2/detection_loss‚1.4697960446566103 %loss/total_loss‚79.55896035631372 trainer/global_step‚49 _runtime‚ 44.569833802²´Ý¾”¯‚(:‰ Ô¯çÒ¯¨é¤ gpu.0.gpu‚90.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…<ë‡ä°‚(:Ù Ô¯çҨ𫧠disk.sda3.out‚ 0.7265625 disk.rbd3.in‚0( proc.memory.availableMB‚ 499.39453125 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2635.828125 cpu‚19.873502979213733 network.recv‚46795 disk.rbd0.out‚0 disk.sda.out‚ 0.7265625 disk.rbd1.out‚0.203125 disk.rbd2.out‚0$ disk./.usageGB‚547.9803161621094 disk.dm-1.out‚ 0.7265625 disk.dm-0.out‚25.28125( disk./.usagePercent‚36.7754188037493 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 7.3203125 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 7.3203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.out‚25.28125 disk.sda3.in‚0 disk.dm-0.in‚ 7.3203125 disk.nvme0n1.out‚25.28125$ memory_percent‚96.95193767547607* proc.memory.percent‚16.087818145751953 proc.cpu.threads‚17 network.sent‚60268 disk.drbd2165.in‚0 disk.sda.in‚0§eÔ””±‚(:‰ ã¯çÒØó£ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚255.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶qH3õ²‚(:ê ã¯çÒÖÈÐ¥ disk.rbd2.in‚0 disk.sda.in‚0 network.sent‚102797$ disk./.usageGB‚547.9804649353027 disk.drbd2165.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.087913513183594 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 1.08203125) disk./.usagePercent‚36.77542878804313 disk.sda1.in‚0% disk.nvme0c0n1.out‚29.82177734375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 7.3203125# disk.nvme0n1.out‚29.82177734375 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 7.3203125 network.recv‚53890 disk.dm-1.in‚0 cpu‚20.73777544473763 disk.rbd2.out‚0 disk.dm-1.out‚ 1.08203125 disk.rbd1.in‚0 disk.rbd1.out‚0.203125 proc.memory.rssMB‚ 2635.84375 disk.dm-0.in‚ 7.3203125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 1.08203125 disk.rbd3.out‚0$ memory_percent‚96.95243835449219 disk.dm-0.out‚29.82177734375 disk.rbd0.in‚0$ proc.memory.availableMB‚499.3125‰v{Ÿ³‚(:” ò¯çÒÌÎÜ£ gpu.0.gpu‚61.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚218.193' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.638600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨vè´‚(:Ý ò¯çÒ¤¾â¥ network.sent‚108368 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚1.28125 disk.dm-0.out‚ 36.90234375 disk.drbd2165.out‚0 disk.rbd0.out‚0 network.recv‚57622) disk./.usagePercent‚36.77541368359861 disk.rbd2.out‚0 disk.nvme0n1.out‚ 36.90234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 8.1484375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚1.28125" memory_percent‚96.952223777771* proc.memory.percent‚16.087937355041504 proc.cpu.threads‚17" disk.nvme0c0n1.out‚ 36.90234375 disk.sda.out‚1.28125 disk.rbd1.in‚0 disk.rbd3.in‚0$ disk./.usageGB‚547.9802398681641 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 8.1484375 disk.sda2.out‚0 disk.rbd1.out‚0.265625( proc.memory.availableMB‚ 499.34765625# proc.memory.rssMB‚ 2635.84765625 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 8.1484375 cpu‚21.404976517075196¥$Ê.µ) _wandbruntime‚84  _runtime‚84sD­Ã¶½ ' lr-AdamW/pg1‚0.00011698538294687907 ' lr-AdamW/pg2‚0.00011698538294687907 trainer/global_step‚99 % _timestamp‚1.7842728927578094e+09 _runtime‚ 84.890475921 _step‚4tŠ.Ç·Á trainer/global_step‚99 % _timestamp‚1.7842728927578094e+09 _runtime‚ 84.890475921 _step‚4 ' lr-AdamW/pg1‚0.00011698538294687907 ' lr-AdamW/pg2‚0.00011698538294687907.wŸ–.¸) _wandbruntime‚84  _runtime‚84òÑàP¢'¹œO VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101982129664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14759943946240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14788515167586 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚213997 _runtime‚ 84.890580949 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚180378 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚767060 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 trainer/global_step‚99 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102323289088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚767060 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14788157579864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14862267235328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14686351003648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14862267235328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3466854729728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3466836117504 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚767062 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3570861648896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚915606 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚905980 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14861909036544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚394378 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14686333964288 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 101823615576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1682668 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14759926906880 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚905980 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚180381 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1673040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0*À^¨rº O \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3570969017856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚767060 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 trainer/global_step‚99 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102323289088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚767060 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14788157579864 F@DeviceStatsMonitor.on_t>@æ4 rain_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14862267235328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14686351003648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14862267235328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3466854729728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3466836117504 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚767062 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3570861648896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚915606 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚905980 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14861909036544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚394378 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14686333964288 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 101823615576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1682668 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14759926906880 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚905980 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚180381 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1673040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101982129664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14759943946240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14788515167586 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚213997 _runtime‚ 84.890580949 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚180378 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 105140354048 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14832844636160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7842728933716278e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚398211 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14907184975872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14935679305458 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚215802 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15010520899072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15010520899072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105264640 trainer/global_step‚99 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3605318577664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚924672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 103335923200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3605423842304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14832861675520 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102994756608 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102994756608 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚397925 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚774692 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3500283488256 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 103175217772 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚915032 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚182123 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14936036893292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455Ü°&½ªM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 103335923200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3605423842304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14832861675520 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102994756608 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102994756608 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774694 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚397925 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚774692 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3500283488256 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 103175217772 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚915032 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚182123 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14936036893292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 105140354048 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14832844636160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7842728933716278e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚398211 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14907184975872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14935679305458 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚215802 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15010520899072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15010520899072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105264640 trainer/global_step‚99 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3605318577664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚924672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268s \q.¾) _wandbruntime‚85  _runtime‚85tåà¼ü¿ö 1loss/SRqqt2/assignment_loss‚2.676633414338214 1loss/FBt1/assignment_loss‚0.31811341154389083 1loss/SRqqt2/detection_loss‚0.9913335119738236 /loss/FRt1/assignment_loss‚4.521248327438221 /loss/FRt2/assignment_loss‚5.334447625574175 2loss/SRqqt1/assignment_loss‚2.9186366513923363 /loss/FRt1/detection_loss‚0.3304784018264188 trainer/global_step‚99 _runtime‚ 85.499331969 _step‚7 /loss/FRt2/detection_loss‚1.5409701537586502 .loss/FBt1/detection_loss‚31.30506742869814 .loss/FBt2/detection_loss‚7.420685761725733 &loss/total_loss‚62.319005428196746 % _timestamp‚1.7842728933739603e+09 1loss/FBt2/assignment_loss‚0.27416546448182905 0loss/SRqqt1/detection_loss‚4.687225275445315 epoch‚0éÿìÀú % _timestamp‚1.7842728933739603e+09 1loss/FBt2/assignment_loss‚0.27416546448182905 0loss/SRqqt1/detection_loss‚4.687225275445315 epoch‚0 1loss/SRqqt2/assignment_loss‚2.676633414338214 1loss/FBt1/assignment_loss‚0.31811341154389083 1loss/SRqqt2/detection_loss‚0.9913335119738236 /loss/FRt1/assignment_loss‚4.521248327438221 /loss/FRt2/assignment_loss‚5.334447625574175 2loss/SRqqt1/assignment_loss‚2.9186366513923363 /loss/FRt1/detection_loss‚0.3304784018264188 trainer/global_step‚99 _runtime‚ 85.499331969 _step‚7 /loss/FRt2/detection_loss‚1.5409701537586502 .loss/FBt1/detection_loss‚31.30506742869814 .loss/FBt2/detection_loss‚7.420685761725733 &loss/total_loss‚62.319005428196746gﱞÁ‚(:“ °çÒŸ²¿¤ gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.747' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚36.14940000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y ¸È÷‚(:ì °çÒ¨‹È¦ disk.sda3.out‚ 1.51171875 disk.dm-1.out‚ 1.51171875 disk.rbd1.in‚0 disk.dm-0.out‚ 45.35546875 disk.drbd2165.in‚0 network.recv‚61704 disk.sda3.in‚0 network.sent‚124859 disk.dm-1.in‚0 disk.rbd1.out‚0.265625 disk.rbd3.out‚0 disk.rbd0.out‚0* disk./.usagePercent‚36.775473333354086# disk./.usageGB‚547.981128692627 disk.sda.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.95284366607666 cpu‚21.920588843410734 disk.sda.out‚ 1.51171875 disk.dm-0.in‚ 8.86328125 disk.rbd2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 8.86328125 disk.sda2.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2635.85546875" disk.nvme0c0n1.out‚ 45.35546875 disk.nvme0n1.out‚ 45.35546875 disk.sda1.in‚0( proc.memory.availableMB‚ 499.24609375* proc.memory.percent‚16.087985038757324 disk.sda2.out‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 8.86328125 disk.rbd0.in‚0ÿ†þ”Â(:‰ °çÒ¶¬¤ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.538' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y´òÄ‚(:ç °çÒù±Î¦ disk.rbd2.out‚0 disk.rbd1.out‚0.34375 disk.dm-0.out‚ 52.412109375 disk.drbd2165.in‚0 disk.sda2.in‚0) disk./.usagePercent‚36.77551378254449 disk.rbd2.in‚0! disk.nvme0n1.out‚ 52.412109375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.out‚1.890625* proc.memory.percent‚16.088032722473145 disk.rbd3.out‚0 disk.sda2.out‚0 network.recv‚66168 disk.nvme0n1.in‚ 9.69140625 disk.dm-1.in‚0 disk.dm-1.out‚1.890625# memory_percent‚96.9557523727417$ disk./.usageGB‚547.9817314147949 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 9.69140625 disk.dm-0.in‚ 9.69140625( proc.memory.availableMB‚ 498.76953125 proc.cpu.threads‚17# disk.nvme0c0n1.out‚ 52.412109375 disk.sda.out‚1.890625 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 network.sent‚161552 disk.drbd2165.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2635.86328125 cpu‚22.322054290827563¬ýÜ©Å‚(:‰ Ÿ°çÒ¶™Å¤ gpu.0.gpu‚59.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temw´~!ëp‚35.0 gpu.0.powerWatts‚114.304' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªŒ8+ÿÆ‚(:ô Ÿ°çÒèõÖ¦% disk.nvme0c0n1.out‚58.44189453125 disk.dm-0.in‚ 9.69140625' proc.memory.availableMB‚ 498.8671875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 9.69140625 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚58.44189453125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.95515632629395# disk.nvme0n1.out‚58.44189453125 disk.sda.out‚ 2.18359375 disk.rbd1.out‚0.34375 proc.cpu.threads‚17 disk.rbd0.out‚0 network.sent‚167121$ disk./.usageGB‚547.9815406799316 disk.sda3.in‚0# proc.memory.rssMB‚ 2635.87109375 cpu‚22.650759205807233 disk.nvme0n1.in‚ 9.69140625 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0* proc.memory.percent‚16.088080406188965* disk./.usagePercent‚36.775500982167785 disk.drbd2165.in‚0 disk.sda3.out‚ 2.18359375 disk.dm-1.out‚ 2.18359375 disk.rbd0.in‚0 network.recv‚70270 disk.rbd2.out‚0FÐK0Ç+ _wandbruntime‚125  _runtime‚125Ì]ÂÞÀȺ % _timestamp‚1.7842729337107973e+09 _runtime‚ 125.836431 _step‚8 & lr-AdamW/pg1‚0.0001760689096877271 & lr-AdamW/pg2‚0.0001760689096877271 trainer/global_step‚149=6k#Äɾ & lr-AdamW/pg2‚0.0001760689096877271 trainer/global_step‚149 % _timestamp‚1.7842729337107973e+09 _runtime‚ 125.836431 _step‚8 & lr-AdamW/pg1‚0.0001760689096877271\§ß¡0Ê+ _wandbruntime‚125  _runtime‚125ÍwM¬'˦O bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 155550765056 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 152677631488 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 _runtime‚ 125.837660341 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22274656019968 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚267628 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚587054 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 153018790912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1148662 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1148662 _step‚9 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109466112 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22164307660372 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152440102484 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22274656019968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚319155 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚586783 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚319423 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22121978388480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5138274043904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22275014218752 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22121995427840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚587054 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 153018790912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1148662 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1148662 _step‚9 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109466112 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22164307660372 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152440102484 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22274656019968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚319155 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚586783 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚319423 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22121978388480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5138274043904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22275014218752 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22121995427840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚10 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1156292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22269219418112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22269236457472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5328190420992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22423267882496 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22423267882496 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 153690258432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1379124 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 trainer/global_step‚149 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2535418 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚590864 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚590588 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5328291491328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚269376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 _runtime‚ 126.450867143 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 101070336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚321488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22269219418112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22158395269120 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22312186973800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22158378229760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22269236457472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 156487618560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1379124 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 156570076672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ED $Áâ$eviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22269219418112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22158395269120 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22312186973800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22158378229760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22269236457472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 156487618560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1379124 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 156570076672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚10 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1156292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22269219418112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22269236457472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5328190420992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22423267882496 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22423267882496 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 153690258432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1379124 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 trainer/global_step‚149 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2535418 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚590864 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚590588 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5328291491328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚269376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 _runtime‚ 126.450867143 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 101070336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚321488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29540457740880 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29687403003392 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚778827 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1814884 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29687761202176 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29687761202176 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚354881 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3354772 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1530260 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203373133312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚13 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 203714292736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29337401151488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 166.711627924 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1824510 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 206815956992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1824510 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6809711970304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1530262 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1814884 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29687403003392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29337418190848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚778546 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29484046909440 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7016527927296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29540457740880 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29687403003392 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚778827 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1814884 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29687761202176 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29687761202176 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚354881 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3354772 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1530260 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203373133312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚13 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 203714292736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29337401151488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 166.711627924 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_starÄò)· t/allocation.small_pool.allocated‚1824510 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 206815956992 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1824510 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6809711970304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1530262 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1814884 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29687403003392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29337418190848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚778546 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29484046909440 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7016527927296 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29631287939072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3361828 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29483911823360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7051082203136 _runtime‚ 167.328976087 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29631287939072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29836014865920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚356623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204385760256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29631270899712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3371470 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204385760256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29483928862720 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204408191588 trainer/global_step‚199 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1537892 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88749568 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6843140728832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537892 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29631270899712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1833576 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚425789 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29688337054308 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚356626 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29835656659968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7050974841344 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 207922862080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7842729752031453e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1537894 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1823936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚14 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1823936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204067643114 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚782412 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 207834112512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1537894 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3361828 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6843140728832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537892 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29631270899712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1833576 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚425789 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29688337054308 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚356626 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29835656659968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7050974841344 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 207922862080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7842729752031453e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1537894 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1823936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚14 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1823936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204067643114 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚782412 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 207834112512 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1537894 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3361828 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29631287939072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3361828 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29483911823360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7051082203136 _runtime‚ 167.328976087 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29631287939072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0\ë…®0â+ _wandbruntime‚167  _runtime‚167h“w@ãú /loss/FRt1/assignment_loss‚4.499038982391357 2loss/SRqqt1/assignment_loss‚2.3363369846343995 %loss/total_loss‚41.90685896375487 epoch‚0 _step‚15 0loss/FBt2/assignment_loss‚0.2592933907028448 0loss/FRt1/detection_loss‚0.31230321471593425 0loss/SRqqt1/detection_loss‚4.241046576094391 _runtime‚ 167.330857138 0loss/FBt1/assignment_loss‚0.2453584843129055 1loss/SRqqt2/detection_loss‚1.0567290315348572 /loss/FBt1/detection_loss‚11.985986028767153 .loss/FBt2/detection_loss‚9.173829852110504 trainer/global_step‚199 0loss/FRt2/assignment_loss‚4.6552447423986765 2loss/SRqqt2/assignment_loss‚1.6902839529428109 /loss/FRt2/detection_loss‚1.4514077231490419 % _timestamp‚1.7842729752054193e+09V:¦Óäþ 0loss/FBt2/assignment_loss‚0.2592933907028448 0loss/FRt1/detection_loss‚0.31230321471593425 0loss/SRqqt1/detection_loss‚4.241046576094391 _runtime‚ 167.330857138 0loss/FBt1/assignment_loss‚0.2453584843129055 1loss/SRqqt2/detection_loss‚1.0567290315348572 /loss/FBt1/detection_loss‚11.985986028767153 .loss/FBt2/detection_loss‚9.173829852110504 trainer/global_step‚199 0loss/FRt2/assignment_loss‚4.6552447423986765 2loss/SRqqt2/assignment_loss‚1.6902839529428109 /loss/FRt2/detection_loss‚1.4514077231490419 % _timestamp‚1.7842729752054193e+09 /loss/FRt1/assignment_loss‚4.499038982391357 2loss/SRqqt1/assignment_loss‚2.3363369846343995 %loss/total_loss‚41.90685896375487 epoch‚0 _step‚15À”lYå‚(:’ Û°çÒ®ƒß¤ gpu.0.gpu‚17.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³LK1ôæ‚(:é Û°çÒª¥¦ disk.sda3.out‚ 3.74609375 disk.rbd3.in‚0 disk.dm-1.in‚0* disk./.usagePercent‚36.775297200170534 disk.rbd2.in‚0 disk.nvme0n1.out‚ 92.05859375 disk.rbd0.out‚0 cpu‚23.546246427614484 disk.dm-0.in‚ 12.77734375 disk.sda2.out‚0 disk.rbd1.out‚0.484375 network.sent‚241701 disk.rbd2.out‚0 disk.sda1.in‚0$ disk./.usageGB‚547.9785041809082 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚96.95806503295898& proc.memory.availableMB‚ 498.390625* proc.memory.percent‚16.088104248046875 disk.dm-1.out‚ 3.74609375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 12.77734375! disk.nvme0c0n1.in‚ 12.77734375" disk.nvme0c0n1.out‚ 92.05859375 proc.memory.rssMB‚2635.875 disk.sda.out‚ 3.74609375 disk.sda3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.recv‚85724 disk.sda.in‚0 disk.dm-0.out‚ 92.05859375 disk.drbd2165.in‚0 disk.rbd0.in‚0æ¹v”ç‚(:‰ ê°çÒßü£ gpu.0.gpu‚88.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚236.204' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.2408 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ²ðè‚(:å ê°çÒˆì¥) disk./.usagePercent‚36.77529464009518$ disk./.usageGB‚547.9784660339355 disk.dm-0.out‚98.40625 disk.sda1.out‚0 network.sent‚278421 disk.dm-0.in‚ 12.77734375 disk.rbd2.out‚0 disk.nvme0n1.in‚ 12.77734375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 3.96484375# proc.memory.rssMB‚ 2635.87890625 proc.cpu.threads‚17 disk.dm-1.out‚ 3.96484375 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.out‚98.40625 disk.nvme0c0n1.out‚98.40625 disk.sda.in‚0 disk.rbd1.out‚0.546875$ memory_percent‚96.94886207580566* proc.memory.percent‚16.088128089904785 cpu‚23.703505982575425 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 499.8984375 network.recv‚90457 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 12.77734375 disk.sda3.out‚ 3.96484375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0°§C0é+ _wandbruntime‚207  _runtime‚207QK?¼Äê¾ _runtime‚ 207.518331522 _step‚16 & lr-AdamW/pg1‚0.0002942359631694232 & lr-AdamW/pg2‚0.0002942359631694232 trainer/global_step‚249 % _timestamp‚1.7842730153927152e+09ï˜ëÈë _runtime‚ 207.518331522 _step‚16 & lr-AdamW/pg1‚0.0002942359631694232 & lr-AdamW/pg2‚0.0002942359631694232 trainer/global_step‚249 % _timestamp‚1.7842730153927152e+09‡éT0ì+ _wandbruntime‚207  _runtime‚2077ïæí¦O OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2269336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1911862 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8739143150592 _step‚17 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚527916 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚442131 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8481149896704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 254409794560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257879593472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 254013624662 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2278962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36846098391040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95048192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528161 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 253673076300 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 254409794560 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37100508185600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 $ _timestamp‚1.784273015393781e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113660416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37100149986816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 257974641664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0F6G°'îªO D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37100149986816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 257974641664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2269336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1911862 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8739143150592 _step‚17 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚527916 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚442131 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8481149896704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 254409794560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257879593472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 254013624662 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2278962 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36846098391040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚95048192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528161 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 253673076300 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 254409794560 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37100508185600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 $ _timestamp‚1.784273015393781e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113660416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268ðB<0ï+ _wandbruntime‚208  _runtime‚208ïtcBµ&ð¯M MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1919494 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 255081262080 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1919492 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8514597267456 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2278388 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36993322381312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚530190 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚443876 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36809462456320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 $ _timestamp‚1.784273016006749e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258997520896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36993339420672 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 255422428672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚974066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 208.13257873 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2288028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8514578655232 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37064129546982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚973817 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1919492 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37248761849344 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37064487134816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36993339420672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36809445416960 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 255024678496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8773594788352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ƒÆžñ³M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚973817 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1919492 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37248761849344 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37064487134816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36993339420672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36809445416960 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 255024678496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8773594788352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1919494 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 255081262080 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1919492 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8514597267456 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2278388 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36993322381312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚530190 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚443876 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36809462456320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 $ _timestamp‚1.784273016006749e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258997520896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36993339420672 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 255422428672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚974066 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 208.13257873 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2288028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8514578655232 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37064129546982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segLÊu« ment.small_pool.current‚268 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44512896970240 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44512896970240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44208149872640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44293115489618 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚632260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 309042780160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2293460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115757568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 309139925504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1161906 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2293460 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305105296384 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1161638 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43988468338688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 304764136960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2293462 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2733414 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚529378 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44513255169024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2723788 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44208132833280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44208132833280 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10461746360832 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10461630603264 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5017248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2733414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10152587823104 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2293462 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44513255169024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5026876 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚632525 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305105296384 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚529381 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10152606435328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44512896970240 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44512896970240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44208149872640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44293115489618 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚632260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 309042780160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2293460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115757568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 309139925504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1161906 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2293460 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305105296384 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1161638 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43988468338688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 304764136960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824õ8šþ0+ _wandbruntime‚248  _runtime‚248:‚g ‚·M ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88749568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2301094 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 306117930496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44355373862912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44661508832768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1165413 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 trainer/global_step‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10496194696704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 305776763904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2742480 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚531123 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2742480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 310070753280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44355390902272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305641165404 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10186016581632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44440279627490 _runtime‚ 248.959856545 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚22 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360œ IüÁ&ƒ»M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44355373862912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚634290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842730568340046e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 305776763904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44440637215324 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚634569 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10496087334912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2301094 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2732840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44355390902272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44661508832768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1165413 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 trainer/global_step‚299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10496194696704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 305776763904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2742480 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚531123 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2742480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 310070753280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44355390902272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305641165404 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10186016581632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44440279627490 _runtime‚ 248.959856545 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚22 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88749568 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2301094 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 306117930496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44355373862912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2675060 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51570184314880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51669265570126 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2675060 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 355800798208 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚616631 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2675062 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11824025749504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 354906050116 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51925643953664 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 355459638784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚737342 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11824044361728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12184293639168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2675062 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51570201354240 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1353705 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚737077 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51570201354240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51925643953664 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 355800798208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 355246598478 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51926002152448 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3178240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 355459638784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51570184314880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12184401008128 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚616628 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11824025749504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 354906050116 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51925643953664 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 355459638784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚737342 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11824044361728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12184293639168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2675062 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51570201354240 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1353705 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚737077 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51570201354240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51925643953664 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 355800798208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 355246598478 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51926002152448 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3178240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 355459638784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51570184314880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12184401008128 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚616628 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2675060 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169E+Ò0“+ _wandbruntime‚289  _runtime‚289d~Zñ½&”·M =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 356813432320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52074255816192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51460512604160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2682694 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚618376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚349 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52073897610240 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5869984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 _step‚26 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51717442383872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2682692 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3187292 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11857473120256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12218733368832 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3187292 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 361278860800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 356472265728 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚739171 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11857454508032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51816429707998 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51717425344512 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 356472265728 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1357544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51816787295832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51717425344512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚739446 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 361375998976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13578222gãÁ&•»M WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12218733368832 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3187292 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 361278860800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 356472265728 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚739171 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11857454508032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51816429707998 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51717425344512 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 356472265728 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1357544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51816787295832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51717425344512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚739446 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 361375998976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1357822 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 356813432320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52074255816192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51460512604160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2682694 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚618376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚349 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52073897610240 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5869984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 _step‚26 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51717442383872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2682692 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3187292 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11857473120256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5879626 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952³eÇÅ0–+ _wandbruntime‚289  _runtime‚289Ç}z»þ—ø /loss/FRt2/assignment_loss‚3.794032268144272 2loss/SRqqt2/assignment_loss‚1.5286675612574911 0loss/FBt1/assignment_loss‚0.2729943249654145 %loss/total_loss‚36.58920439184082 % _timestamp‚1.7842730976581554e+09 _runtime‚ 289.783503863 /loss/FRt1/detection_loss‚0.4620409511326524 /loss/FRt2/detection_loss‚1.3922739197728082 epoch‚0 2loss/SRqqt1/assignment_loss‚1.8593438420001631 /loss/FBt2/detection_loss‚14.122427371708122 /loss/FRt1/assignment_loss‚4.072288825683387 1loss/FBt2/assignment_loss‚0.17362073668695105 /loss/SRqqt1/detection_loss‚3.57259690973582 1loss/SRqqt2/detection_loss‚0.8565731480972417 .loss/FBt1/detection_loss‚4.482344532656499 trainer/global_step‚349 _step‚27g7­a˜ü /loss/FRt1/detection_loss‚0.4620409511326524 /loss/FRt2/detection_loss‚1.3922739197728082 epoch‚0 2loss/SRqqt1/assignment_loss‚1.8593438420001631 /loss/FBt2/detection_loss‚14.122427371708122 /loss/FRt1/assignment_loss‚4.072288825683387 1loss/FBt2/assignment_loss‚0.17362073668695105 /loss/SRqqt1/detection_loss‚3.57259690973582 1loss/SRqqt2/detection_loss‚0.8565731480972417 .loss/FBt1/detection_loss‚4.482344532656499 trainer/global_step‚349 _step‚27 /loss/FRt2/assignment_loss‚3.794032268144272 2loss/SRqqt2/assignment_loss‚1.5286675612574911 0loss/FBt1/assignment_loss‚0.2729943249654145 %loss/total_loss‚36.58920439184082 % _timestamp‚1.7842730976581554e+09 _runtime‚ 289.783503863*KÍžž™‚(:“ Ó±çÒùÿÁ¤ gpu.0.gpu‚44.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ¸Æ%š‚(:ê Ó±çÒÈŽ»¦ network.“4¾˜Ðsent‚399385 disk.sda3.out‚ 6.08984375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda.in‚0 cpu‚24.42222893439745 disk.rbd1.out‚0.828125 disk.rbd3.in‚0 disk.dm-0.out‚ 162.267578125 disk.rbd0.in‚0* proc.memory.percent‚16.088247299194336 disk.sda2.in‚0 disk.rbd3.out‚0" disk.nvme0n1.out‚ 162.267578125 disk.nvme0c0n1.in‚17.40625$ memory_percent‚96.95467948913574) disk./.usagePercent‚36.77538245067943 disk.rbd2.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚119623$ disk./.usageGB‚547.9797744750977 disk.sda2.out‚0 disk.dm-0.in‚17.40625 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 162.267578125' proc.memory.availableMB‚ 498.9453125 disk.dm-1.out‚ 6.08984375 disk.nvme0n1.in‚17.40625 disk.sda.out‚ 6.08984375" proc.memory.rssMB‚ 2635.8984375qMŸ›‚(:” â±çҲ欥 gpu.0.gpu‚88.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.408' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.281600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 1ÿœ‚(:û â±çÒ€Þ¥§ disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0! disk.dm-0.out‚171.55615234375$ disk.nvme0n1.out‚171.55615234375 disk.sda1.out‚0 disk.nvme0n1.in‚ 18.12109375 disk.sda.in‚0( proc.memory.availableMB‚ 499.41796875* proc.memory.percent‚16.088247299194336 network.recv‚124348$ disk./.usageGB‚547.9789123535156 disk.rbd1.in‚0 disk.rbd1.out‚0.890625 disk.dm-0.in‚ 18.12109375 disk.drbd2165.out‚0 disk.sda.out‚ 6.26171875 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 6.26171875 proc.cpu.threads‚17 network.sent‚449018 disk.rbd0.in‚0" proc.memory.rssMB‚ 2635.8984375! disk.nvme0c0n1.in‚ 18.12109375& disk.nvme0c0n1.out‚171.55615234375 cpu‚24.48566910891993 disk.sda3.out‚ 6.26171875* disk./.usagePercent‚36.775324592976695 disk.rbd2.out‚0 disk.sda2.in‚0$ memory_percent‚96.95179462432861«Ž„°0+ _wandbruntime‚329  _runtime‚329ÂÍ*£ÆžÀ _step‚28 ' lr-AdamW/pg1‚0.00047148654339196724 ' lr-AdamW/pg2‚0.00047148654339196724 trainer/global_step‚399 % _timestamp‚1.7842731378717418e+09 _runtime‚ 329.998588412ÀB°ÊŸÄ _runtime‚ 329.998588412 _step‚28 ' lr-AdamW/pg1‚0.00047148654339196724 ' lr-AdamW/pg2‚0.00047148654339196724 trainer/global_step‚399 % _timestamp‚1.7842731378717418e+09Ž,90 + _wandbruntime‚329  _runtime‚329À´®N³'¡­O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 405863085386 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58932235796480 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3056660 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3642318 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1545670 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58639552565248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 % _timestamp‚1.7842731378738222e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3056662 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59338390937088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13906880005120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 406155140608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 405522537024 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6698980 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59045058062912 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3632692 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58932252835840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59338749135872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13495463675904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3642318 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 406496300032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚841789 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99242496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 _step‚29 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚703878 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3056660 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59338390937088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58639535525888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6689352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13495482288128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 411416329216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚399 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3632692 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58932252835840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 411515571712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59045415650634 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚841507 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13906997859840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59338749135872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117854720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1545385 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 406496300032 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 406155140608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58932235796480 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 411416329216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚399 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3632692 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58932252835840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 411515571712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59045415650634 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚841507 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13906997859840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59338749135872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117854720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1545385 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 406496300032 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 406155140608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58932235796480 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3056660 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3642318 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1545670 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58639552565248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 % _timestamp‚1.7842731378738222e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3056662 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59338390937088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13906880005120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 406155140608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 405522537024 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6698980 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59045058062912 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3632692 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58932252835840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59338749135872íF©í’£‚(:‡ ñ±çÒëèî¤ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S-R¤‚(:õ ñ±çÒó½î§ network.sent‚452479! disk.nvme0c0n1.in‚ 18.94921875* disk./.usagePercent‚36.775325617006835 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 18.94921875 disk.dm-1.in‚0$ disk./.usageGB‚547.9789276123047# disk.nvme0n1.out‚183.5966796875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 6.5390625 cpu‚24.546743176680877 disk.rbd2.out‚0% disk.nvme0c0n1.out‚183.5966796875 proc.cpu.threads‚17 network.recv‚128357 disk.dm-0.out‚183.5966796875 disk.rbd2.in‚0 disk.sda.out‚ 6.5390625 disk.sda3.in‚0$ memory_percent‚96.95329666137695 disk.rbd1.out‚0.890625 disk.rbd3.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2635.90234375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda3.out‚ 6.5390625& proc.memory.availableMB‚ 499.171875 disk.nvme0n1.in‚ 18.94921875 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.088271141052246@ #õ0¥+ _wandbruntime‚330  _runtime‚330*S·¦¶M F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6706036 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3064294 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3064292 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 407167767552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59487002799616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59486644593664 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚705626 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98973184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3064292 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 407508934144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13941446478848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1549482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6706036 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 412455071232 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 406874139220 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1549233 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59192937376340 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3641744 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚843610 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13528892434432 % _timestamp‚1.7842731384963028e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3064294 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13528911046656 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59192579788506 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59079493865472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 trainer/global_step‚399 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 F‡µ«*¨@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 _runtime‚ 330.622122026 _step‚30 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59487002799616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80360960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 406533590746 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3651384 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3641744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58786063237120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58786046197760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246š‹˜lÀ&§ºM WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13941347505664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3651384 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 407167767552 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚843856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13941446478848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1549482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6706036 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 412455071232 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 406874139220 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1549233 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59192937376340 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3641744 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚843610 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13528892434432 % _timestamp‚1.7842731384963028e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3064294 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13528911046656 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59192579788506 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59079493865472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 trainer/global_step‚399 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 _runtime‚ 330.622122026 _step‚30 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59487002799616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80360960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 406533590746 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3651384 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3641744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58786063237120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58786046197760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6706036 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3064294 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3064292 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 407167767552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59487002799616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59486644593664 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚705626 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98973184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3064292 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 407508934144 E5H·bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPþÒZžJ¸bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP{'ÐN¹bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPMHºbC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP“Ÿ?×N»bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP 'ŠN¼bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÎþóyH½bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPý>KJ¾bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPßÙGæJ¿bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÿŽe,JÀbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP€5Ÿ¤NÁbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPábÍ;Âb6 loss/val_total_loss"trainer/global_step2:JP2P5JÃbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPE>ŠJÄbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP[Ž†HÅbC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPUŸ9JÆbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŠVÖÄ;Çb6 jet/accuracy_3_of_4"trainer/global_step2:JPÞ­àeNÈbI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPî‹÷ JÉbE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPô©ÁHÊbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPùÕtóJËbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP>€cêHÌbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP•JNTNÍbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPçzïí@Îb; particle/accuracy_2_of_3"trainer/global_step2:JPI®ÚHÏbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPVc±JÐbE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP'9q¥JÑbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPYïQNÒbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP˜¹¡JÓbE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPµú¡œJÔbE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPä˜vÔJÕbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP£,$*JÖbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPh÷:çJ×bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP I;BNØbI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP…~‰ÖJÙbE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPžô"NÚbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPõÓ?§NÛbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPE··9NÜbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP œe0NÝbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP±øHÞbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPX-)HßbC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP8 KYNàbI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP-;áb6 jet/accuracy_1_of_3"trainer/global_step2:JP¼˜Sn8âb3 particle/f_score"trainer/global_step2:JPV{ÚJãbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPˆ h;äb6 jet/accuracy_1_of_4"trainer/global_step2:JPÞ¤;;åb6 jet/accuracy_4_of_4"trainer/global_step2:JP]仉@æb; particle/accuracy_1_of_1"trainer/global_step2:JPcrJJçbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÇǸiNèbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPɵ4’JébE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÙikÓHêbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP®^zNëbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPzoLë@ìb; particle/accuracy_3_of_4"trainer/global_step2:JPˆ›Ü;íb6 jet/accuracy_5_of_5"trainer/global_step2:JP?SMHîbC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP™ÚnNïbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP¨BHðbC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP¶»¢JñbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP4 ¯HòbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPu> »;ób6 jet/accuracy_4_of_5"trainer/global_step2:JPæjQNôbI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPá’HõbC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPøl›@öb; particle/accuracy_4_of_4"trainer/global_step2:JPB/ ¿H÷bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPvô¸ÃNøbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPj]ùNùbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPS+ÎJúbE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP¾ÎÀEJûbE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPâ‹&Š@üb; particle/accuracy_2_of_5"trainer/global_step2:JP Õ[è@ýb; particle/accuracy_5_of_5"trainer/global_step2:JPfÈ·CJþbE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPŸ:JÿbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPžæž1@€b; particle/accuracy_3_of_5"trainer/global_step2:JPá6ÅJbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPE'rªH‚bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPæ÷/ŠNƒbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP¡©ÒH„bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPEŠ§ÎJ…bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPS‰ˆäH†bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP÷–›ÓJ‡bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÑÄ6/JˆbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÝ;o(;‰b6 jet/accuracy_3_of_3"trainer/global_step2:JPô,¬œHŠbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPëÚaN‹bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPš—Æ‹NŒbI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPEAÜ HbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP­Ú{NŽbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPógÓHbC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP°%±HbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPnŽJÛ@‘b; particle/accuracy_2_of_2"trainer/global_step2:JPEË "@’b; particle/accuracy_1_of_4"trainer/global_step2:JP S˜d;“b6 jet/accuracy_2_of_5"trainer/global_step2:JPhÉÖH”bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPòÅJ•bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPµTàJ–bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPM²@ J—bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP=át»H˜bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPwäÀèJ™bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP©Á€KNšbI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÉG J›bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPX©ç¯JœbE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP7BHbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPygFHžbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP¦¿ªJŸbE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP:ÉRH bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPL¿¶N¡bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPH XH¢bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPÏO»8H£bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÜéÏŽN¤bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP#Ï N¥bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP7c¾H¦bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPecòûN§bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP‹æŒÂ;¨b6 jet/accuracy_2_of_3"trainer/global_step2:JPš^Äw;©b6 jet/accuracy_2_of_4"trainer/global_step2:JPÈ´HªbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP-i©ÅJ«bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP ŒDN¬bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPM†J­bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP1EÇDJ®bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP¯Ÿë N¯bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPVÎÓH°bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP : ÆH±bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPõñ›~H²bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPœ–®Y;³b6 jet/accuracy_1_of_5"trainer/global_step2:JPÝ ¯N´bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPªÏ‡:HµbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPŒA–ùJ¶bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP>òÀJ·bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPu‡×N¸bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPWªH¹bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPøе;ºb6 jet/accuracy_3_of_5"trainer/global_step2:JPÎó/@»b; particle/accuracy_4_of_5"trainer/global_step2:JP)ÛÊùH¼bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÅ`1H½bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPh&@†N¾bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP¯ aN¿bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÛ<‚HÀbC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP"&*–JÁbE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP<ÛÏãJÂbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¸žNÃbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPˆæÂJÄbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÁCC‘HÅbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP+)¿nHÆbC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPvçÏkHÇbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPþg.ùGÈbB validation_average_jet_accuracy"trainer/global_step2:JPÀ³ÎýHÉbC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPTLHÊbC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPüSÑJËbE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP®hU!JÌbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPæX¶HÍbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÇýÃ…JÎbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÞËHÏbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPU×yVJÐbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP— ®¶;Ñb6 jet/accuracy_1_of_1"trainer/global_step2:JP³Ë(JÒbE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÄ46JÓbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP‘`¿aJÔbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPb á˜JÕbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPË!ü-JÖbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPyD?N×bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÅ\^JØbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÞ§„9JÙbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPh HÚbC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP5ÁpHÛbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÈ:Ž8JÜbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPˆ‚JÝbE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPªÊ@JÞbE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPLõPÜHßbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP@ÝTJàbE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP4ïÒØNábI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¾kPJâbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÏtʵJãbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP´È·†NäbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¨,R¼NåbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPò=û JæbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP±Ç¶çNçbI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP¯W&HèbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPw–ãêJébE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPVØLHêbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÄI"JëbE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPN3ò@ìb; particle/accuracy_1_of_2"trainer/global_step2:JPí^˜NíbI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP£×´HîbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPã8i/JïbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPə٬;ðb6 jet/accuracy_2_of_2"trainer/global_step2:JP¸Ÿo½JñbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP­ît*JòbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPñžšŠJóbE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPèºHôbC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPØÈHõbC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP«Xð¾JöbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP’TH÷bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP˜ôJøbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP“} JùbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP½“ÄHúbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP‚5œHûbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÉwÓÛJübE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP0z–ãHýbC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPäðNþbI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP£¬°`JÿbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÆÌ -J€bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÕ u×HbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPîiUµJ‚bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPZ>HƒbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP.ž<ÏH„bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP5÷‡«J…bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP_ª%íN†bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPT ÒÉH‡bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPN ’Ã;ˆb6 jet/accuracy_1_of_2"trainer/global_step2:JP}¡=îN‰bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÓuAHŠbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP“Å{˜H‹bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP7ü…yHŒbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:J ô5Pã³4çJbE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPæoäHŽbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPå$ïRNbI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPvâïæ@b; particle/accuracy_2_of_4"trainer/global_step2:JP㙌H‘bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPæDÅsN’bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP×2£iH“bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPûåRòN”bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP6§ç“N•bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPà2áJ–bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÆþp‡N—bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP±D(=N˜bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP4hø”J™bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÂÿ NšbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPe(vN›bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPµÎS'JœbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPoÍÁNbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÊÖÿJžbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP#÷SÎJŸbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP’žÓ.J bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPJ°ç¶J¡bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP8‚eJ¢bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPócrH£bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPduƽJ¤bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP—É›J¥bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÁeFJ¦bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPP=uJ§bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPG5ó N¨bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP:=|H©bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPSüÞ‚NªbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP9«ämN«bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPª„‰J¬bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPä 8N­bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPŒ€þ[H®bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPh|ëH¯bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP)mqÝH°bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP×£•J±bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPks¼N²bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPôÍX1N³bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP„~‰ž@´b; particle/accuracy_1_of_5"trainer/global_step2:JP!HJµbE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP¿'H¶bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPrîLƒN·bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÔ79N¸bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP¬‘ù@¹b; particle/accuracy_3_of_3"trainer/global_step2:JP¶ËĬNºbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP”y/J»bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÁgÿH¼bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP îÇJ½bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPäÜ"[<¾b7 particle/sensitivity"trainer/global_step2:JPðª0Q<¿b7 particle/specificity"trainer/global_step2:JPì@Àb; particle/accuracy_1_of_3"trainer/global_step2:JP'¼‰HÁbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP`/ŸJÂbE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP´@OJÃbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPH‰…HÄbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP‹ñe¬HÅbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPG„ÆHÆbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP«qsHÇbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPüiA´HÈbC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPpò×hHÉbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÜžUkHÊbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP „%9Ëb4 particle/accuracy"trainer/global_step2:JPk µHÌbC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP’æŠ0Í+ _wandbruntime‚349  _runtime‚349þ4Äí0>Ϊ| ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.15498867630958557 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15577659010887146 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.11938110738992691 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.07201137393712997 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.11716616898775101 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.08671431988477707 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.018322931602597237 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.015698250383138657 *jet/accuracy_1_of_2‚0.3625743091106415 trainer/global_step‚399 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9456565380096436 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.16738474369049072 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.39392998814582825 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1562577486038208 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9814751744270325 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.6153846383094788 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.19248555600643158 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0012210012646391988 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.14142486453056335 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1886661946773529 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.008737517520785332 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12055050581693649 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1348532885313034 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6203703284263611 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6368072628974915 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8344373106956482 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt0SRqqt1FBt/event_purity‚0.029819373041391373 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.5925925374031067 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13485506176948547 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9823157787322998 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.1254270374774933 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19679491221904755 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9917027354240417 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /particle/accuracy_3_of_3‚0.9985215067863464 % _timestamp‚1.7842731572248333e+09 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.012499568983912468 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.4707108438014984 *particle/specificity‚0.866025447845459 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9050700664520264 *"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.18482504785060883 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9681479334831238 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07534771412611008 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0640726312994957 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.18133032321929932 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.5555555820465088 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.01020987331867218 'particle/accuracy‚0.781122088432312 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9777016639709473 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9172186255455017 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4221475124359131 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07630068063735962 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.1242971271276474 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.18838998675346375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06301157176494598 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.1265636384487152 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.09942483901977539 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.11343544721603394 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07698412984609604 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0022330323699861765 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *loss/val_total_loss‚26.500558793935223 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.012154800817370415 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9655386805534363 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08975182473659515 jet/accuracy_3_of_4‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.12590391933918 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9616355895996094 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009999999776482582 9 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.013888888992369175 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9621145129203796 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7445359230041504 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.23156006634235382 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.13428500294685364 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02083333395421505 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.1849871575832367 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.01679614931344986 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9831295013427734 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_3‚0.5381578803062439 'particle/f_score‚0.4801591634750366 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.1254270374774933 *jet/accuracy_1_of_4‚0.5244488716125488 jet/accuracy_4_of_4‚0 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5400680303573608 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6111111044883728 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_4‚0.9861055016517639 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9583067893981934 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9690764546394348 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.431410551071167 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9823618531227112 /particle/accuracy_4_of_4‚0.8608757257461548 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7548434138298035 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.0790729895234108 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.015461363829672337 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.008981375023722649 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.13883274793624878 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.07473819702863693 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.07880545407533646 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.42694681882858276 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.004098047036677599 _runtime‚ 349.351173693 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1634920835494995 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24035599827766418 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 .particle/accuracy_2_of_2‚0.999891996383667 particle/accuracy_1_of_4‚1 jet/accuracy_2_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.0777193233370781 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.10964903235435486 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.1286555975675583 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.41981446743011475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1926196813583374 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07403608411550522 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.8996422290802002 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9823157787322998 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9163737297058105 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23535247147083282 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_3‚0.13326752185821533 *jet/accuracy_2_of_4‚0.1255100518465042 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2254745215177536 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7868631482124329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.01262911967933178 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0873015895485878 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.973320484161377 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09487706422805786 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1339259296655655 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.15120823681354523 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9538702368736267 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09487706422805786 jet/accuracy_3_of_5‚0 particle/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9653093218803406 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21970687806606293 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.11317799240350723 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0020055482164025307 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.13755537569522858 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9788730144500732 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9852380156517029 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.12296687066555023 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2109375 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.02066851779818535 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.22499999403953552 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12550221383571625 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.11336106061935425 +jet/accuracy_1_of_1‚0.08993539959192276 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.0382634736597538 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.00391502445563674 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2430555522441864 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6111111044883728 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.15120823681354523 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6203703284263611 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.09419345110654831 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.03500000014901161 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.5956854820251465 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08735605329275131 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9270310401916504 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.15035361051559448 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0006613756413571537 _step‚32 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.003994571976363659 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 epoch‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07421940565109253 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0494791679084301 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.016155891120433807 particle/accuracy_1_of_2‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9503306150436401 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.1049409732222557 +jet/accuracy_2_of_2‚0.04273848608136177 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.40450650453567505 *"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.01786918193101883 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9521624445915222 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.07727674394845963 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.016155891120433807 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.005602241028100252ÃÁj½u-Ï®| :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.40450650453567505 *"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.01786918193101883 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9521624445915222 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.07727674394845963 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.016155891120433807 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.005602241028100252 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.15498867630958557 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15577659010887146 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.11938110738992691 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.07201137393712997 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.11716616898775101 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.08671431988477707 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.018322931602597237 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.42307692766189575 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.015698250383138657 *jet/accuracy_1_of_2‚0.3625743091106415 trainer/global_step‚399 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9456565380096436 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.16738474369049072 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.39392998814582825 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1562577486038208 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9814751744270325 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.6153846383094788 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.19248555600643158 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0012210012646391988 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.14142486453056335 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1886661946773529 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.008737517520785332 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12055050581693649 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1348532885313034 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6203703284263611 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.42307692766189575 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6368072628974915 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8344373106956482 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt0SRqqt1FBt/event_purity‚0.029819373041391373 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.5925925374031067 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13485506176948547 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9823157787322998 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.1254270374774933 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19679491221904755 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9917027354240417 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /particle/accuracy_3_of_3‚0.9985215067863464 % _timestamp‚1.7842731572248333e+09 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.012499568983912468 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.4707108438014984 *particle/specificity‚0.866025447845459 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9050700664520264 *"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.18482504785060883 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9681479334831238 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07534771412611008 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0640726312994957 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.18133032321929932 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.5555555820465088 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.01020987331867218 'particle/accuracy‚0.781122088432312 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9777016639709473 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9172186255455017 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4221475124359131 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07630068063735962 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.1242971271276474 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.18838998675346375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.06301157176494598 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.1265636384487152 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.09942483901977539 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.11343544721603394 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07698412984609604 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0022330323699861765 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *loss/val_total_loss‚26.500558793935223 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.012154800817370415 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9655386805534363 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08975182473659515 jet/accuracy_3_of_4‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.12590391933918 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9616355895996094 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009999999776482582 9 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.013888888992369175 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9621145129203796 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7445359230041504 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.23156006634235382 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.13428500294685364 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02083333395421505 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.1849871575832367 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.01679614931344986 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9831295013427734 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_3‚0.5381578803062439 'particle/f_score‚0.4801591634750366 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.1254270374774933 *jet/accuracy_1_of_4‚0.5244488716125488 jet/accuracy_4_of_4‚0 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5400680303573608 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6111111044883728 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 /particle/accuracy_3_of_4‚0.9861055016517639 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9583067893981934 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9690764546394348 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.431410551071167 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9823618531227112 /particle/accuracy_4_of_4‚0.8608757257461548 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7548434138298035 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.0790729895234108 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.015461363829672337 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.008981375023722649 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.13883274793624878 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.07473819702863693 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.07880545407533646 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.42694681882858276 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.004098047036677599 _runtime‚ 349.351173693 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1634920835494995 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24035599827766418 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 .particle/accuracy_2_of_2‚0.999891996383667 particle/accuracy_1_of_4‚1 jet/accuracy_2_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.0777193233370781 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.10964903235435486 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.1286555975675583 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.41981446743011475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1926196813583374 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07403608411550522 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.8996422290802002 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9823157787322998 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9163737297058105 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00020926339493Á¶þw¿59879 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23535247147083282 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_3‚0.13326752185821533 *jet/accuracy_2_of_4‚0.1255100518465042 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2254745215177536 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7868631482124329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.01262911967933178 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.0873015895485878 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.973320484161377 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09487706422805786 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1339259296655655 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.15120823681354523 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9538702368736267 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09487706422805786 jet/accuracy_3_of_5‚0 particle/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9653093218803406 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21970687806606293 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.11317799240350723 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0020055482164025307 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.13755537569522858 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9788730144500732 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9852380156517029 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.12296687066555023 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2109375 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.02066851779818535 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.22499999403953552 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12550221383571625 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.11336106061935425 +jet/accuracy_1_of_1‚0.08993539959192276 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.0382634736597538 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.00391502445563674 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2430555522441864 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6111111044883728 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.15120823681354523 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6203703284263611 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.09419345110654831 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.03500000014901161 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.5956854820251465 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08735605329275131 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9270310401916504 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.15035361051559448 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0006613756413571537 _step‚32 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.003994571976363659 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 epoch‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07421940565109253 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0494791679084301 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.016155891120433807 particle/accuracy_1_of_2‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9503306150436401 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.1049409732222557 +jet/accuracy_2_of_2‚0.04273848608136177 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 |e›’Ђ(:‡ ²çÒž›ñ¢ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èIÞ±ÿÑ‚(:ô ²çÒ—ί network.recv‚137098 disk.sda.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚96.97983264923096 disk.rbd2.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 494.82421875) proc.memory.percent‚16.61350727081299 disk.rbd2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2721.95703125$ disk./.usageGB‚548.4591979980469 disk.sda2.out‚0 disk.rbd1.out‚0.96875! disk.dm-0.out‚233.91357421875 cpu‚24.77042790899739 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚7.078125 disk.sda3.out‚7.078125 disk.dm-1.out‚7.078125* disk./.usagePercent‚36.807556965567684 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 19.67578125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 network.sent‚526534 disk.nvme0n1.in‚ 19.67578125$ disk.nvme0n1.out‚233.91357421875& disk.nvme0c0n1.out‚233.91357421875 disk.dm-0.in‚ 19.67578125Eß½Ò‚(:… ž²çÒîéÀ¢ gpu.0.gpu‚99.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚234.9' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.98 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒƒònÓ‚(:õ ž²çÒ¿óŸ¦ disk.sda.out‚ 7.4296875 network.sent‚584487 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 20.50390625 disk.sda1.in‚0 disk.sda3.out‚ 7.4296875$ disk./.usageGB‚548.4591102600098 disk.rbd0.in‚0( proc.memory.percent‚16.6135311126709 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 7.4296875 disk.dm-0.in‚ 20.50390625 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 20.50390625 disk.rbd2.out‚0) disk./.usagePercent‚36.80755107739439 network.recv‚141551 disk.rbd1.out‚1.09375 disk.drbd2165.out‚0 disk.sda2.out‚0! disk.dm-0.out‚700.16357421875$ disk.nvme0n1.out‚700.16357421875" proc.memory.rssMB‚ 2721.9609375 cpu‚24.817771203336246& disk.nvme0c0n1.out‚700.16357421875$ memory_percent‚96.97949886322021( proc.memory.availableMB‚ 494.87890625£§X žÔ‚(:“ ­²çÒ®´â¤ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óoÍœáÕ‚(:Ö ­²çÒÚ¾ã¦# proc.memory.availableMB‚494.875 disk.nvme0n1.in‚21.21875 disk.nvme0c0n1.in‚21.21875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚21.21875" proc.memory.rssMB‚ 2721.9609375 cpu‚24.860719464660647* disk./.usagePercent‚36.807557477582755$ disk./.usageGB‚548.4592056274414 disk.sda3.out‚7.625 disk.rbd1.in‚0" disk.nvme0n1.out‚ 709.173828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 709.173828125 disk.rbd0.out‚0$ memory_percent‚96.97952270507812 network.sent‚592088 disk.sda2.in‚0 disk.dm-1.out‚7.625 disk.rbd1.out‚1.09375 network.recv‚145332 disk.rbd2.out‚0 disk.sda.out‚7.625 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 709.173828125 disk.sda.in‚0( proc.memory.percent‚16.6135311126709 disk.drbd2165.out‚0 disk.rbd3.out‚0Õè‹0Ö+ _wandbruntime‚391  _runtime‚391fw›^Æ×À ' lr-AdamW/pg1‚0.00047266092482630887 ' lr-AdamW/pg2‚0.00047266092482630887 trainer/global_step‚449 % _timestamp‚1.7842731995409904e+09 _runtime‚ 391.670576558 _step‚33€TìÊØÄ ' lr-AdamW/pg1‚0.00047266092482630887 ' lr-AdamW/pg2‚0.00047266092482630887 trainer/global_step‚449 % _timestamp‚1.7842731995409904e+09 _runtime‚ 391.670576558 _step‚33!‰¿G0Ù+ _wandbruntime‚391  _runtime‚391ÓþR²'Ú¬O RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚976128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 _step‚34 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67873732333568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4167152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚976381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473858748001 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68698437628416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68698796005888 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 480690852352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68223853928448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4157177 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 474925038080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4157177 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15182813702144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 480764750336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 474583699968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15663597064704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚793648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 474199297883 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68223870967808 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92510208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4167152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1770032 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68347948670811 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68347591081569 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67873749372928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 474925038080 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 _runtime‚ 391.672668827?Í6–¶'Û°O [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67873732333568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4167152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚976381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473858748001 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68698437628416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68698796005888 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 480690852352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68223853928448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4157177 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 474925038080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4157177 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15182813702144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 480764750336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 474583699968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15663597064704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚793648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 474199297883 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68223870967808 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92510208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4167152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1770032 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68347948670811 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68347591081569 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67873749372928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 474925038080 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 _runtime‚ 391.672668827 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚976128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 _step‚34"¦~ j0Ü+ _wandbruntime‚392  _runtime‚392«$µ2° ݵM J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68020243005440 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 475596326912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 481741648384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68495112807163 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 trainer/global_step‚449 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68371094958080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68371111997440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1773855 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15698070320640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚978219 _runtime‚ 392.285943878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1773612 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.curren U† t‚86211584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15697984109056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 475937672192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4166229 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68846691284992 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚35 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4166229 $ _timestamp‚1.784273200160109e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4176218 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚795393 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68495470396517 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3566560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68371111997440 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4176218 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 475210351717 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68847049669632 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚795396 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566560 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 481809247744 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3566558 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚978459 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68020243005440 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 475596326912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 481741648384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68495112807163 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 trainer/global_step‚449 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68371094958080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68371111997440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1773855 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15698070320640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚978219 _runtime‚ 392.285943878 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1773612 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚86211584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15697984109056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 475937672192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4166229 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68846691284992 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚35 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4166229 $ _timestamp‚1.784273200160109e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4176218 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚795393 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68495470396517 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3566560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68371111997440 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4176218 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 475210351717 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68847049669632 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚795396 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566560 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 481809247744 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3566558 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77# c‡Ñ0ß+ _wandbruntime‚392  _runtime‚392jiÁ´àü % _timestamp‚1.7842732001623998e+09 _runtime‚ 392.287727966 /loss/FRt2/assignment_loss‚2.996247267742373 1loss/SRqqt1/assignment_loss‚1.684180638741657 1loss/SRqqt2/detection_loss‚0.8439574859186948 epoch‚1 2loss/SRqqt2/assignment_loss‚1.4243986410613214 1loss/FBt2/assignment_loss‚0.12204868149962508 /loss/FRt2/detection_loss‚1.5229242161659007 0loss/SRqqt1/detection_loss‚2.793928870341736 1loss/FBt1/assignment_loss‚0.17798298296284706 /loss/FBt1/detection_loss‚2.2911095483182753 &loss/total_loss‚24.043140483387766 _step‚36 0loss/FRt1/assignment_loss‚4.0043252418575355 /loss/FRt1/detection_loss‚0.5327824127446721 /loss/FBt2/detection_loss‚5.6492544960331275 trainer/global_step‚449¸`ëœá€ 1loss/FBt2/assignment_loss‚0.12204868149962508 /loss/FRt2/detection_loss‚1.5229242161659007 0loss/SRqqt1/detection_loss‚2.793928870341736 1loss/FBt1/assignment_loss‚0.17798298296284706 /loss/FBt1/detection_loss‚2.2911095483182753 &loss/total_loss‚24.043140483387766 _step‚36 0loss/FRt1/assignment_loss‚4.0043252418575355 /loss/FRt1/detection_loss‚0.5327824127446721 /loss/FBt2/detection_loss‚5.6492544960331275 trainer/global_step‚449 % _timestamp‚1.7842732001623998e+09 _runtime‚ 392.287727966 /loss/FRt2/assignment_loss‚2.996247267742373 1loss/SRqqt1/assignment_loss‚1.684180638741657 1loss/SRqqt2/detection_loss‚0.8439574859186948 epoch‚1 2loss/SRqqt2/assignment_loss‚1.4243986410613214$q·)’â‚(:‡ ¼²çÒ¬ÜÝ£ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì×þã‚(:ó ¼²çÒüú›§ disk.sda3.in‚0 disk.rbd1.out‚1.171875! disk.dm-0.out‚716.62939453125 disk.drbd2165.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 494.18359375 cpu‚24.898599517989886 proc.cpu.threads‚17) disk./.usagePercent‚36.80740540910741$ disk./.usageGB‚548.4569396972656 disk.dm-0.in‚ 22.046875 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 22.046875" proc.memory.rssMB‚ 2721.9609375 network.recv‚149235 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 22.046875$ disk.nvme0n1.out‚716.62939453125 network.sent‚608447 disk.sda3.out‚ 7.97265625 disk.dm-1.out‚ 7.97265625 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚716.62939453125 disk.sda.in‚0 disk.sda.out‚ 7.97265625$ memory_percent‚96.98374271392822( proc.memory.percent‚16.6135311126709 disk.sda2.in‚0a♟ä‚(:” ˲çÒ×£š¤ gpu.0.gpu‚76.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01üå‚(:ñ ˲çÒÂÄü¥ disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 22.046875 disk.sda.in‚0 cpu‚24.93476154978753# disk./.usageGB‚548.456958770752 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2721.9609375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 8.19921875 disk.dm-1.out‚ 8.19921875 disk.dm-0.in‚ 22.046875! disk.dm-0.out‚723.04931640625$ memory_percent‚96.98188304901123) disk./.usagePercent‚36.80740668914508 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚1.171875( proc.memory.availableMB‚ 494.48828125 proc.cpu.threads‚17 disk.sda.out‚ 8.19921875 network.sent‚658631 disk.nvme0n1.in‚ 22.046875 disk.rbd3.out‚0 network.recv‚154452 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk.nvme0n1.out‚723.04931640625 disk.rbd0.out‚0& disk.nvme0c0n1.out‚723.04931640625 disk.dm-1.in‚0 disk.rbd3.in‚0( proc.memory.percent‚16.6135311126709èç0æ+ _wandbruntime‚432  _runtime‚432ÓŠPMÄç¾ _runtime‚ 432.696442474 _step‚37 & lr-AdamW/pg1‚0.0004726384599318108 & lr-AdamW/pg2‚0.0004726384599318108 trainer/global_step‚499 % _timestamp‚1.7842732405667577e+09•¤ÌÈè & lr-AdamW/pg1‚0.0004726384599318108 & lr-AdamW/pg2‚0.0004726384599318108 trainer/global_step‚499 % _timestamp‚1.7842732405667577e+09 _runtime‚ 432.696442474 _step‚37%@}Üß0é+ _wandbruntime‚432  _runtime‚432u>2àµ'ê¯O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75585922449408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚880898 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76111184611840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1079761 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17386332474368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 524815784791 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75585905410048 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 525279201792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16854270240768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940526 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76111542989312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75585905410048 _runtime‚ 432.696541411 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76111542989312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 532167326720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 _step‚38 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4621604 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4611629 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1960938 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3940526 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚880901 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 532080845824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17386437567488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75724098751319 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76111184611840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75723741162077 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4611629 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3940528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 525279201792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75585922449408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75199282966528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75724098751319 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76111184611840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75723741162077 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4611629 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3940528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 525279201792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75585922449408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75199282966528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 524815784791 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75585905410048 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 525279201792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16854270240768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3940526 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76111542989312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75585905410048 _runtime‚ 432.696541411 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76111542989312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 532167326720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 _step‚38 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4621604 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4611629 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1960938 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576&>íÕh0ì+ _wandbruntime‚433  _runtime‚433=…Åu¾&í¸M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 525826838625 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4620681 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17420894022144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚499 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 533195022848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75733146439680 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8568839 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚882646 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1964753 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17420799421952 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 533119034880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3948158 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75345793638400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3948158 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 526633174016 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 526291828736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76259796653056 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1082107 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75733163479040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 525486288631 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 526291828736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94600192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4630670 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75345776599040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚39 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 _runtime‚ 433.312429329 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1964509 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16887698999296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360“«ãÏÂ&î¼M FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94600192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4630670 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75345776599040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚39 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 _runtime‚ 433.312429329 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1964509 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16887698999296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 525826838625 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4620681 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17420894022144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚499 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 533195022848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75733146439680 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8568839 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚882646 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1964753 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17420799421952 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 533119034880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3948158 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75345793638400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3948158 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 526633174016 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 526291828736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76259796653056 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1082107 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75733163479040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 525486288631 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 526291828736'‘ð^Ó0ï+ _wandbruntime‚433  _runtime‚433¯ã¢üðö epoch‚1 % _timestamp‚1.7842732411886857e+09 1loss/SRqqt1/assignment_loss‚1.712696591343779 /loss/FRt1/detection_loss‚0.5847881338224781 .loss/FBt1/detection_loss‚1.114168296016338 .loss/FBt2/detection_loss‚4.380152911096811 _runtime‚ 433.314024 0loss/FRt2/assignment_loss‚2.9902940523552606 2loss/SRqqt2/assignment_loss‚1.3118028099022128 /loss/FRt2/detection_loss‚1.3859072537219814 1loss/SRqqt1/detection_loss‚2.7486801365172706 /loss/FRt1/assignment_loss‚3.858379972066811 1loss/FBt1/assignment_loss‚0.11347730294131289 1loss/FBt2/assignment_loss‚0.23762675619538612 trainer/global_step‚499 _step‚40 0loss/SRqqt2/detection_loss‚0.777461693744408 %loss/total_loss‚21.21543590972405lráñú 1loss/SRqqt1/assignment_loss‚1.712696591343779 /loss/FRt1/detection_loss‚0.5847881338224781 .loss/FBt1/detection_loss‚1.114168296016338 .loss/FBt2/detection_loss‚4.380152911096811 _runtime‚ 433.314024 0loss/FRt2/assignment_loss‚2.9902940523552606 2loss/SRqqt2/assignment_loss‚1.3118028099022128 /loss/FRt2/detection_loss‚1.3859072537219814 1loss/SRqqt1/detection_loss‚2.7486801365172706 /loss/FRt1/assignment_loss‚3.858379972066811 1loss/FBt1/assignment_loss‚0.11347730294131289 1loss/FBt2/assignment_loss‚0.23762675619538612 trainer/global_step‚499 _step‚40 0loss/SRqqt2/detection_loss‚0.777461693744408 %loss/total_loss‚21.21543590972405 epoch‚1 % _timestamp‚1.7842732411886857e+09(`<Õžò‚(:“ Ú²çÒÇ gpu.0.gpu‚93.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.30499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0× Æ÷ó‚(:ì Ú²çÒ“Òç! disk.nvme0c0n1.in‚ 22.76171875 disk.rbd3.out‚0) disk./.usagePercent‚36.80742332963481 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚0$ memory_percent‚96.98140621185303" disk.nvme0c0n1.out‚ 729.7578125 disk.dm-1.out‚ 8.48046875( proc.memory.percent‚16.6135311126709 network.sent‚677121 disk.sda1.out‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2721.9609375 proc.cpu.threads‚17 disk.dm-0.in‚ 22.76171875 network.recv‚158102 disk.nvme0n1.in‚ 22.76171875 disk.nvme0n1.out‚ 729.7578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 8.48046875 disk.rbd2.in‚0$ disk./.usageGB‚548.4572067260742 disk.rbd0.in‚0 disk.rbd1.out‚1.234375 disk.rbd3.in‚0 cpu‚24.96680347583392 disk.sda.out‚ 8.48046875 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.dm-0.out‚ 729.7578125( proc.memory.availableMB‚ 494.56640625”ÿ‹žŸô‚(:” é²çÒžïê¤ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚194.181' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.836200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>}ù³õ‚(:í é²çÒé’ú¦ disk.rbd1.out‚1.234375 disk.dm-0.in‚ 23.58984375$ memory_percent‚96.98257446289062# proc.memory.availableMB‚494.375 disk.rbd2.out‚0ZgCÝE disk.sda.in‚0 disk.sda.out‚ 8.66796875 disk.rbd1.in‚0 disk.dm-0.out‚ 739.501953125! disk.nvme0c0n1.in‚ 23.58984375 disk.rbd3.out‚0) disk./.usagePercent‚36.80744713833549$ disk./.usageGB‚548.4575614929199" disk.nvme0n1.out‚ 739.501953125 disk.sda1.in‚0( proc.memory.percent‚16.6135311126709 cpu‚25.00066125124542 disk.rbd2.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 739.501953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2721.9609375 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.recv‚163086 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚727328 disk.nvme0n1.in‚ 23.58984375 disk.sda1.out‚0 disk.sda3.out‚ 8.66796875 disk.dm-1.out‚ 8.66796875Lc@%žö‚(:“ ø²çÒᤤ gpu.0.gpu‚41.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;•\ðú÷‚(:ï ø²çÒ»€©¦ disk.sda.in‚0 disk.sda.out‚ 8.79296875 disk.dm-1.out‚ 8.79296875 disk.rbd1.out‚1.3125 disk.dm-0.in‚ 24.3046875! disk.dm-0.out‚746.02001953125 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 24.3046875$ memory_percent‚96.98257446289062 proc.cpu.threads‚17) disk./.usagePercent‚36.80751165223412 disk.drbd2165.out‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2721.9609375 disk.nvme0n1.in‚ 24.3046875$ disk.nvme0n1.out‚746.02001953125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.availableMB‚494.375 disk.rbd2.out‚0 disk.sda3.out‚ 8.79296875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 network.sent‚732741 network.recv‚166479$ disk./.usageGB‚548.4585227966309 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 cpu‚25.028617184795927& disk.nvme0c0n1.out‚746.02001953125 disk.sda3.in‚0( proc.memory.percent‚16.6135311126709£Aä0ø+ _wandbruntime‚473  _runtime‚473å3õÅù¿ ' lr-AdamW/pg1‚0.00047260081765029225 ' lr-AdamW/pg2‚0.00047260081765029225 trainer/global_step‚549 $ _timestamp‚1.784273281628891e+09 _runtime‚ 473.754475382 _step‚41Q5¦Éúà ' lr-AdamW/pg1‚0.00047260081765029225 ' lr-AdamW/pg2‚0.00047260081765029225 trainer/global_step‚549 $ _timestamp‚1.784273281628891e+09 _runtime‚ 473.754475382 _step‚41) \o¦0û+ _wandbruntime‚473  _runtime‚473­˜­1µ'ü¯O G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚533 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5066081 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82524799520768 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83523931595264 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83523931595264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 583335174144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19109024729088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5076056 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 575974703616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82947973931008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2151269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 575974703616 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4322128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9388207 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1183382 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82524816560128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82947973931008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842732816299694e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82947956891648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚968151 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19109129822208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18525689554944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83100248831827 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322126 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83099891242585 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5076056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82947956891648 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83523931595264 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83523931595264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 583335174144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19109024729088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5076056 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 575974703616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82947973931008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2151269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 575974703616 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4322128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9388207 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1183382 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82524816560128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82947973931008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842732816299694e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82947956891648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚968151 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19109129822208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18525689554944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83100248831827 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322126 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83099891242585 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5076056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82947956891648 HDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2155350 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9414882 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82671310192640 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 576443325533 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83672543636480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚969893 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83672185251840 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚533 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329758 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 577328675840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5075133 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚468 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83247412968179 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4329758 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842732822427962e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83672543636480 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83095197921280 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19143470651904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚549 _runtime‚ 474.368657017 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 576987330560 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 577328675840 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 576987330560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5085122 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83095214960640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1185203 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 584352338432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5085122 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚969896 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19143584126464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82671327232000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2155096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5075133 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 584447200768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83095214960640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18559136925696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚8324777055753±Â÷èu3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 O¼Ã&€½M HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94862336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2155350 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9414882 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82671310192640 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 576443325533 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83672543636480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚969893 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83672185251840 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚533 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329758 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 577328675840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5075133 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚468 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83247412968179 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4329758 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842732822427962e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83672543636480 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83095197921280 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19143470651904 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚549 _runtime‚ 474.368657017 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 576987330560 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 577328675840 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 576987330560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5085122 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83095214960640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1185203 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 584352338432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5085122 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚969896 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19143584126464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82671327232000 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2155096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5075133 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 584447200768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83095214960640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18559136925696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83247770557533 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 Op¸ž„‚(:“ ‡³çÒÝÔÓ¤ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.603' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.12060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼"ØÜ÷…‚(:ì ‡³çÒ¸Ò˦ disk.sda3.in‚0" proc.memory.rssMB‚ 2721.9609375 cpu‚25.05584121656193 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.out‚1.3125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 24.3046875 disk.drbd2165.out‚0 disk.dm-0.out‚ 752.041015625 disk.nvme0c0n1.in‚ 24.3046875 disk.sda2.out‚0 disk.sda.out‚ 9.19140625 disk.dm-1.in‚0 disk.dm-1.out‚ 9.19140625( proc.memory.percent‚16.6135311126709 proc.cpu.threads‚17* disk./.usagePercent‚36.807500899917684 disk.dm-0.in‚ 24.3046875$ memory_percent‚96.98247909545898 network.sent‚738300" disk.nvme0n1.out‚ 752.041015625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 9.19140625 disk.rbd1.in‚0& proc.memory.availableMB‚ 494.390625$ disk./.usageGB‚548.4583625793457 disk.rbd2.out‚0$ disk.nvme0c0n1.out‚ 752.041015625 disk.rbd3.in‚0 network.recv‚170412Ù„Ú’†‚(:‡ –³çÒ¡ˆ­¥ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿE¦Eõ‡‚(:ê –³çÒÁ¾Ý§ disk.rbd2.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 759.056640625& proc.memory.availableMB‚ 493.203125 disk.rbd1.out‚1.375 network.recv‚175180 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.08562953868285 disk.dm-1.in‚0 disk.dm-0.in‚ 25.1328125( proc.memory.percent‚16.6135311126709) disk./.usagePercent‚36.80750909215878$ disk./.usageGB‚548.4584846496582 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 25.1328125$ memory_percent‚96.98972702026367 disk.sda3.in‚0 disk.sda3.out‚ 9.69140625 disk.dm-1.out‚ 9.69140625 network.sent‚801554 disk.rbd3.in‚0 disk.nvme0n1.in‚ 25.1328125 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2721.9609375 disk.rbd1.in‚0" disk.nvme0n1.out‚ 759.056640625 disk.rbd0.out‚0 disk.sda.out‚ 9.69140625 disk.dm-0.out‚ 759.056640625!{Dôžˆ‚(:“ ¥³çÒÒÀ¤ gpu.0.gpu‚34.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01p·ø‰‚(:í ¥³çÒÈñû¦ disk.sda.in‚0 disk.sda3.out‚ 9.8515625 disk.dm-0.in‚ 25.84765625" proc.memory.rssMB‚ 2721.9609375 cpu‚25.111290300109413$ memory_percent‚96.98693752288818 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 25.84765625( proc.memory.availableMB‚ 493.66015625( proc.memory.percent‚16.6135311126709 network.recv‚178371 disk.rbd2.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 768.701171875 disk.dm-1.out‚ 9.8515625) disk./.usagePercent‚36.80753059679166 disk.drbd2165.in‚0 disk.sda.out‚ 9.8515625 disk.rbd1.out‚1.375 disk.sda2.out‚0 proc.cpu.threads‚17" disk.nvme0n1.out‚ 768.701171875 disk.sda1.out‚0 disk.dm-0.out‚ 768.701171875 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 25.84765625 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 network.sent‚808124$ disk./.usageGB‚548.4588050842285¥i00Š+ _wandbruntime‚514  _runtime‚514¨²ñÞÅ‹¿ ' lr-AdamW/pg1‚0.00047254800039944663 ' lr-AdamW/pg2‚0.00047254800039944663 trainer/global_step‚599 % _timestamp‚1.7842733223687747e+09 _runtime‚ 514.49834269 _step‚45ÙYk_Ɍà ' lr-AdamW/pg2‚0.00047254800039944663 trainer/global_step‚599 % _timestamp‚1.7842733223687747e+09 _runtime‚ 514.49834269 _step‚45 ' lr-AdamW/pg1‚0.00047254800039944663-ß_‘õ0+ _wandbruntime‚514  _runtime‚514؇Ѡº'Ž´O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1286541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111384576 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1055398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2341942 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4703726 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2341677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20831727881216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634489015296 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5530508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784273322376274e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4703728 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634581787648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10234236 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92772352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703726 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 626048758607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 625708208725 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90936678578688 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 626670205440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20831616496640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 626670205440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90476398912335 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20197146093568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89850333114368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10224259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90310008373248 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20197127481344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5530508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5520533 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90476398912335 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20197146093568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89850333114368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10224259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90310008373248 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20197127481344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5530508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5520533 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2341677 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20831727881216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634489015296 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5530508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784273322376274e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4703728 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634581787648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10234236 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚599 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92772352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703726 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 626048758607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 625708208725 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90936678578688 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 626670205440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352.1—ž0+ _wandbruntime‚515  _runtime‚515?¤8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5529585 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 635597559808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2345473 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4711358 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20866172411904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20866065228800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4711360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 627682832384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 635508988928 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1288594 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 626719262447 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5539574 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 628024177664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4711358 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1057143 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20230556239872¢7#¬É&’ÃM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1057146 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 627059812441 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 628024177664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5529585 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 635597559808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2345473 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10240943 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4711358 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20866172411904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20866065228800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4711360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 627682832384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 635508988928 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1288594 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 626719262447 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5539574 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 628024177664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4711358 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1057143 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20230556239872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91084932235264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90457266442240 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90623563048687 _runtime‚ 515.117833238 % _timestamp‚1.7842733229919841e+09 _step‚47 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90457249402880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91085290619904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989/±,%0“+ _wandbruntime‚515  _runtime‚515ØŒ…”ý _runtime‚ 515.119347206 _step‚48 0loss/FRt2/assignment_loss‚2.6697220058198186 1loss/FBt2/assignment_loss‚0.14308928344733782 /loss/FRt2/detection_loss‚1.4128329100990615 1loss/SRqqt2/detection_loss‚0.7589613098588806 /loss/FBt2/detection_loss‚2.5788515772550338 &loss/total_loss‚18.035193075735073 1loss/SRqqt1/assignment_loss‚1.428910087532131 2loss/SRqqt2/assignment_loss‚1.2677435173586542 1loss/FBt1/assignment_loss‚0.16238772063063364 /loss/FRt1/detection_loss‚0.5720706849025573 1loss/SRqqt1/detection_loss‚2.2383200004179447 /loss/FBt1/detection_loss‚1.2250660696151616 trainer/global_step‚599 $ _timestamp‚1.784273322993998e+09 0loss/FRt1/assignment_loss‚3.5772379087978567 epoch‚1X1q/• 1loss/SRqqt2/detection_loss‚0.7589613098588806 /loss/FBt2/detection_loss‚2.5788515772550338 &loss/total_loss‚18.035193075735073 1loss/SRqqt1/assignment_loss‚1.428910087532131 2loss/SRqqt2/assignment_loss‚1.2677435173586542 1loss/FBt1/assignment_loss‚0.16238772063063364 /loss/FRt1/detection_loss‚0.5720706849025573 1loss/SRqqt1/detection_loss‚2.2383200004179447 /loss/FBt1/detection_loss‚1.2250660696151616 trainer/global_step‚599 $ _timestamp‚1.784273322993998e+09 0loss/FRt1/assignment_loss‚3.5772379087978567 epoch‚1 _runtime‚ 515.119347206 _step‚48 0loss/FRt2/assignment_loss‚2.6697220058198186 1loss/FBt2/assignment_loss‚0.14308928344733782 /loss/FRt2/detection_loss‚1.41283291009906150y9§Û”–‚(:‰ ´³çÒÖÖ¬¤ gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚237.057' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.4114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þHmñ—‚(:æ ´³çÒÒž¦ disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚548.4588165283203 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 26.67578125 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚1.453125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 26.67578125! disk.nvme0n1.out‚ 776.68359375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.dm-0.out‚ 776.68359375$ memory_percent‚96.98843955993652 network.recv‚182208 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 26.67578125' proc.memory.availableMB‚ 493.4140625" proc.memory.rssMB‚ 2721.9609375( proc.memory.percent‚16.6135311126709) disk./.usagePercent‚36.80753136481426 disk.sda.out‚10.0625 disk.dm-1.out‚10.0625 cpu‚25.136913902386734 network.sent‚811477 disk.rbd2.out‚0# disk.nvme0c0n1.out‚ 776.68359375 disk.sda2.out‚0 disk.sda3.out‚10.0625 disk.rbd0.out‚0 disk.sda.in‚0Æk‚—Ÿ˜‚(:” óçÒêÓ¤ gpu.0.gpu‚52.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ¼f¢õ™‚(:ê óçÒ­Ú¦¦ network.recv‚187304 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0% proc.memory.availableMB‚ 493.21875) disk./.usagePercent‚36.80754032507796$ disk./.usageGB‚548.4589500427246" disk.nvme0n1.out‚ 786.005859375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚1.453125$ memory_percent‚96.98963165283203 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 10.5546875 network.sent‚876878 disk.dm-0.out‚ 786.005859375 disk.nvme0c0n1.in‚ 27.390625 disk.sda.out‚ 10.5546875 cpu‚25.162005787869425 disk.rbd2.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda3.out‚ 10.5546875 disk.dm-0.in‚ 27.390625" proc.memory.rssMB‚ 2721.9609375 disk.rbd2.in‚0( proc.memory.percent‚16.6135311126709 disk.nvme0n1.in‚ 27.390625$ disk.nvme0c0n1.out‚ 786.005859375 disk.sda2.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17¶Þoø”š‚(:‰ Ò³çÒ惤 gpu.0.gpu‚40.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%L†Ž›‚(:õ Ò³çÒ©Ä‹¦ network.sent‚880196 network.recv‚190862 disk.dm-1.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚548.4587898254395 disk.rbd1.out‚1.453125 disk.dm-0.in‚ 27.390625 disk.nvme0n1.in‚ 27.390625 disk.sda.in‚0" proc.memory.rssMB‚ 2721.9609375 disk.dm-1.out‚ 10.83203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 27.390625( proc.memory.percent‚16.6135311126709 cpu‚25.18253851677496 proc.cpu.threads‚17 disk.rbd2.out‚0$ disk.nvme0n1.out‚789.26416015625' proc.memory.availableMB‚ 493.4296875* disk./.usagePercent‚36.807529572761524 disk.rbd1.in‚0! disk.dm-0.out‚789.26416015625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 10.83203125 disk.rbd3.out‚0$ memory_percent‚96.98834419250488 disk.rbd3.in‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚789.26416015625 disk.sda.out‚ 10.83203125DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98349425562112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97852548992843 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1389690 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97852191403601 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5085326 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5974985 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5085326 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 677707045376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98349783939584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 677365707264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 JBDeviceStatsMonitor.ŸyæÃon_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 676665245515 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97672076894208 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5085328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21868565407744 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97672059854848 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5974985 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22554212454912 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5984960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98349783939584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97175883747328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685733528064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 _step‚50 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1142651 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5984960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚649 % _timestamp‚1.7842733633377419e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5085328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2532604 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98349425562112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97852548992843 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1389690 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97852191403601 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5085326 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5974985 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5085326 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 677707045376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98349783939584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 677365707264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 676665245515 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97672076894208 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5085328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21868565407744 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97672059854848 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5974985 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22554212454912 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5984960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98349783939584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97175883747328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685733528064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 _step‚50 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1142651 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5984960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚649 % _timestamp‚1.7842733633377419e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5085328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2532604 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1391972 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5092958 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97819317923840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 _runtime‚ 556.078142754 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98498037603328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97322377379840 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5994026 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1391722 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11076995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5092960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21901994166272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5984037 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678719679488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 686759049216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5092960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 677335749355 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5984037 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2536115 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1144393 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚51 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152IàÒkf¤ÃM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98498037603328 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98497679218688 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2536368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 686674672640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97819300884480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 678719679488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97819317923840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 _runtime‚ 556.078142754 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98498037603328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97322377379840 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5994026 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1391722 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11076995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5092960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21901994166272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5984037 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678719679488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 686759049216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5092960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 677335749355 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5984037 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2536115 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1144393 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚51 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1391972 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5092958 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105762172545536 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 728402547200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105034128375808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104501417340928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5466926 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105762172545536 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 736897041920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚699 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466928 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726941182541 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105762530923008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728061209088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24276918988288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23540003334144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105034111336448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105228699073351 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 727281732423 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 _step‚54 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6439412 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6429437 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2722813 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24276801312256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105762172545536 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 728402547200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105034128375808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104501417340928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5466926 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105762172545536 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 736897041920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚699 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466928 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 726941182541 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105762530923008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728061209088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24276918988288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23540003334144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105034111336448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105228699073351 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 727281732423 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 _step‚54 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6439412 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6429437 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2722813 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24276801312256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 728292786257 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6448478 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 729415181312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727952236263 % _timestamp‚1.7842734047889805e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2726857 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90668032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105181369405440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729073836032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 trainer/global_step‚699 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 737820151808 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24311252244480 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 737910819840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2726584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1231646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1231643 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6438489 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729415181312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105181369405440 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104647910973440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5474558 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105375863209703 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5474558 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105910784586752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 244045578249t’Ô&´ÎM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104647928012800 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23573450704896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 737820151808 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24311252244480 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 737910819840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2726584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1231646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1231643 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6438489 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 729415181312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105181369405440 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104647910973440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5474558 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105375863209703 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5474558 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105910784586752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5474560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105181352366080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6438489 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23573432092672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 728292786257 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6448478 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 729415181312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727952236263 % _timestamp‚1.7842734047889805e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2726857 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90668032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105181369405440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729073836032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 trainer/global_step‚699 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111826950934528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 777898219331 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1596072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778756710912 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5848526 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112604491564617 TDDeviceStatsMonitor.on_train_batch_start/reserv ‚ƒ¦ed_bytes.all.allocated‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6883889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5848528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113175277906432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25211441260544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5848528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6883889 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113174919528960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392“ßË3Ê'ÃÄO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113175277906432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842734451769066e+09 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2913479 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111826933895168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778756710912 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5848526 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112604491564617 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6883889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5848528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113175277906432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25211441260544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5848528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6883889 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113174919528960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96704512 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25999451414528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25211459872768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112604849153859 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25999548119040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚58 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 788088246272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111826950934528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 777898219331 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1596072 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112543420887040 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6902930 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 780110683136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2917003 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1318893 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12749099 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6892941 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 789027789312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 778909273165 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778568723171 trainer/global_step‚749 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25244888631296 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779769337856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5856160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112752370879565 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12759090 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5856158 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2917274 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 780110683136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 _runtime‚ 637.91751341 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1598378 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5856160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1318896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111973444567040 _step‚59 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779769337856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5856160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112752370879565 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12759090 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5856158 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2917274 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 780110683136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 _runtime‚ 637.91751341 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1598378 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5856160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1318896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111973444567040 _step‚59 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779769337856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112543420887040 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6902930 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 780110683136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2917003 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1318893 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12749099 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6892941 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 789027789312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 778909273165 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778568723171 trainer/global_step‚749 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25244888631296 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779769337856;LE å0Ç+ _wandbruntime‚637  _runtime‚637»»½H[Èú $ _timestamp‚1.784273445793733e+09 0loss/FRt1/assignment_loss‚3.4291614726168653 2loss/SRqqt1/assignment_loss‚1.4570912275901613 _step‚60 1loss/FBt2/assignment_loss‚0.15660220646761108 /loss/FRt1/detection_loss‚0.6397628988199884 1loss/SRqqt2/detection_loss‚0.7763990454114533 -loss/FBt2/detection_loss‚2.50531047453051 &loss/total_loss‚16.641568862928423 /loss/FRt2/assignment_loss‚2.438325702710585 1loss/FBt1/assignment_loss‚0.19695717433263832 /loss/FRt2/detection_loss‚1.5014642443474044 1loss/SRqqt1/detection_loss‚1.6354299039525144 .l‹ÃÒ¤¥oss/FBt1/detection_loss‚0.822001719497743 epoch‚1 trainer/global_step‚749 _runtime‚ 637.919058265 2loss/SRqqt2/assignment_loss‚1.0830627926509475&ÁÇ/Éþ _step‚60 1loss/FBt2/assignment_loss‚0.15660220646761108 /loss/FRt1/detection_loss‚0.6397628988199884 1loss/SRqqt2/detection_loss‚0.7763990454114533 -loss/FBt2/detection_loss‚2.50531047453051 &loss/total_loss‚16.641568862928423 /loss/FRt2/assignment_loss‚2.438325702710585 1loss/FBt1/assignment_loss‚0.19695717433263832 /loss/FRt2/detection_loss‚1.5014642443474044 1loss/SRqqt1/detection_loss‚1.6354299039525144 .loss/FBt1/detection_loss‚0.822001719497743 epoch‚1 trainer/global_step‚749 _runtime‚ 637.919058265 2loss/SRqqt2/assignment_loss‚1.0830627926509475 $ _timestamp‚1.784273445793733e+09 0loss/FRt1/assignment_loss‚3.4291614726168653 2loss/SRqqt1/assignment_loss‚1.4570912275901613<°€êŸÊ‚(:” ¬´çÒœóæ¤ gpu.0.gpu‚45.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚158.929' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.785800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÂpm Ë‚(:ÿ ¬´çÒ·òã§" proc.memory.rssMB‚ 2721.9609375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 14.55078125 disk.rbd0.in‚0 disk.sda.out‚ 14.55078125 disk.sda2.in‚0! disk.dm-0.out‚848.25634765625 network.sent‚ 1.041425e+06$ disk./.usageGB‚548.4642066955566 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚99.95274543762207 disk.rbd1.out‚1.65625 disk.dm-0.in‚ 31.30078125$ disk.nvme0n1.out‚848.25634765625 disk.rbd0.out‚0& disk.nvme0c0n1.out‚848.25634765625 disk.rbd1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚36.807893103460145 disk.sda3.out‚ 14.55078125 disk.rbd3.out‚0( proc.memory.percent‚16.6135311126709 cpu‚25.292939277120034 network.recv‚217329 disk.rbd3.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 7.7421875! disk.nvme0c0n1.in‚ 31.30078125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 31.30078125 –£””Ì‚(:‰ »´çÒן¹¤ gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.411' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8822 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿'¹áüÍ‚(:ñ »´çÒ©¿¬¦ disk.sda2.in‚0 disk.sda3.out‚ 16.30078125 disk.rbd1.out‚1.734375 disk.dm-0.out‚ 864.767578125 disk.drbd2165.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 32.015625 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚36.80829861939438 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 32.015625 disk.dm-1.out‚ 16.30078125 network.sent‚ 1.091691e+06$ disk./.usageGB‚548.4702491760254 disk.sda1.out‚0 disk.dm-1.in‚0$ proc.memory.availableMB‚7.796875" disk.nvme0n1.out‚ 864.767578125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 32.015625 proc.cpu.threads‚17 disk.sda1.in‚0 network.recv‚222279 disk.rbd3.out‚0" proc.memory.rssMB‚ 2721.9609375 cpu‚25.30559207575419 disk.rbd2.out‚0$ disk.nvme0c0n1.out‚ 864.767578125 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚99.95241165161133( proc.memory.percent‚16.6135311126709 disk.sda.out‚ 16.30078125i#ǸŸÎ‚(:” Ê´çÒÀº›¥ gpu.0.gpu‚82.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AdöÏ‚(:ë Ê´çґ𥧠disk.rbd0.out‚0 disk.rbd3.in‚0$ proc.memory.availableMB‚7.796875 disk.dm-1.out‚ 16.8359375 cpu‚25.321365673965197 disk.nvme0n1.in‚32.84375 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 network.recv‚225523& disk./.usagePercent‚36.80815371913 disk.nvme0c0n1.in‚32.84375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 16.8359375 disk.rbd2.out‚0" proc.memory.rssMB‚ 2721.9609375 network.sent‚ 1.098916e+06 disk.rbd0.in‚0% disk.nvme0c0n1.out‚876.1240234375 disk.sda3.in‚0 disk.sda3.out‚ 16.8359375( proc.memory.percent‚16.6135311126709# disk./.usageGB‚548.468090057373 disk.sda2.in‚0 disk.rbd1.out‚1.734375 disk.dm-0.in‚32.84375 disk.rbd2.in‚0 disk.sda1.in‚0$ memory_percent‚99.95241165161133 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-0.out‚876.1240234375# disk.nvme0n1.out‚876.1240234375 disk.sda1.out‚0!ºø0Ð+ _wandbruntime‚678  _runtime‚678°ÿÆÆÑÀ ' lr-AdamW/pg1‚0.00047218506416474526 ' lr-AdamW/pg2‚0.00047218506416474526 trainer/global_step‚799 % _timestamp‚1.7842734861137807e+09 _runtime‚ 678.244425104 _step‚61†õ”ÊÒÄ ' lr-AdamW/pg1‚0.00047218506416474526 ' lr-AdamW/pg2‚0.00047218506416474526 trainer/global_step‚799 % _timestamp‚1.7842734861137807e+09 _runtime‚ 678.244425104 _step‚61=Žb1C0Ó+ _wandbruntime‚678  _runtime‚678SMæ«Ç'ÔÁO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107190272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 839249943040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 829793550848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13568467 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6230126 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 829793550848 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 829452212736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119152467488768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13578444 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119980999234367 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 trainer/global_step‚799 _runtime‚ 678.244517729 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1699108 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1698836 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7348316 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119758231339008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7338341 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3103234 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6230128 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7348316 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1404401 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120587666512384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88578048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6230128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119980641645125 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 839338521088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26882897799168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 828514706239 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27722129129984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119152484528128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 _step‚62 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119758231339008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1404398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120587666512384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88578048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6230128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119980641645125 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 839338521088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26882897799168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 828514706239 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27722129129984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119152484528128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 _step‚62 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119758231339008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1404398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 839249943040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 829793550848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13568467 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6230126 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 829793550848 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 829452212736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119152467488768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13578444 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119980999234367 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 trainer/global_step‚799 _runtime‚ 678.244517729 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1699108 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1698836 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7348316 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119758231339008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7338341 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3103234 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6230128 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7348316 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1404401 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169>>BÝT0Ö+ _wandbruntime‚678  _runtime‚678€ÜÐÆg×ËM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1700864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120128163370719 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119905472368640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3107007 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120128520960073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7357382 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7347393 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1406146 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 830464839680 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚63 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119298995200000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120735920168960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 _runtime‚ 678.856931419 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 trainer/global_step‚799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120736278553600 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7357382 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1701131 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27756685799424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842734867310777e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3107277 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 MADeviceStatsMonitor.on_train_bat×!ûj ch_end/active.large_pool.allocated‚6237760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119905455329280 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6237758 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 830806184960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6237758 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27756582810624 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 840274865152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120736278553600 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 830464839680 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 829185210079 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26916307945472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119905455329280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6237760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119298978160640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119905472368640d:õmÕ&ØÏM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7347393 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7357382 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1701131 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27756685799424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842734867310777e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3107277 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6237760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119905455329280 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6237758 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 830806184960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6237758 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27756582810624 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 840274865152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120736278553600 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 830464839680 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 829185210079 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26916307945472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119905455329280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6237760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119298978160640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119905472368640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1700864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120128163370719 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119905472368640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3107007 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120128520960073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7357382 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7347393 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1406146 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 830464839680 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚63 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119298995200000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120735920168960 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 _runtime‚ 678.856931419 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 trainer/global_step‚799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120736278553600 E&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5213876962661743 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06075487285852432 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.26980650424957275 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.42468637228012085 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10138918459415436 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.44300320744514465 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3533293604850769 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47835230827331543 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006349206669256091 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1610439121723175 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *loss/val_total_loss‚15.525659529837057 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9931291937828064 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 /particle/accuracy_3_of_3‚0.9990123510360718 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04264017567038536 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_4‚0.34754660725593567 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4418976604938507 9 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.058150313794612885 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5073750615119934 particle/accuracy_1_of_2‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005248802248388529 % _timestamp‚1.7842735053643408e+09 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.797224223613739 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08172374963760376 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6064608693122864 particle/accuracy_1_of_1‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.01918800175189972 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4844214916229248 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.01949610747396946 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4418976604938507 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.01949610747396946 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.01549927145242691 +jet/accuracy_2_of_3‚0.27307969331741333 particle/accuracy_2_of_2‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8536737561225891 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2568112909793854 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6198035478591919 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3646792769432068 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9131760597229004 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.35229483246803284 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.45666664838790894 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15829935669898987 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04345620423555374 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851877093315125 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2604166865348816 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.018708787858486176 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8274613618850708 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.40007323026657104 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12000000476837158 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.05185672640800476 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5876806378364563 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.471001535654068 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4063183069229126 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961584806442261 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15829935669898987 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6537302136421204 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42400041222572327 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35185185074806213 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972054362297058 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4119214117527008 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5208333730697632 particle/accuracy_2_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.982336699962616 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 epoch‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.42518553137779236 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_4‚1 jet/accuracy_4_of_5‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.30558475852012634 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.996457576751709 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940527677536011 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.36433401703834534 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4631331264972687 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5103174448013306 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924328923225403 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.04631427675485611 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.43367084860801697 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017240718007087708 .particle/accuracy_3_of_4‚0.989321768283844 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.031395573168992996 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0362953320145607 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.03185763955116272 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.47587522864341736 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0012210012646391988 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.42643070220947266 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.38200193643569946 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06552673131227493 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.048396866768598557 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.389187753200531 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/f_score‚0.5655713081359863 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _step‚65 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9951465129852295 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47513702511787415 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.38509315252304077 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010342216119170189 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.7673367857933044 ,jet/accuracy_4_of_4‚0.007130124606192112 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3823491632938385 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5115395784378052 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4383530914783478 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09518452733755112 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3671875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.02506919763982296 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.029035761952400208 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931998252868652 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895977973937988 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4475761950016022 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.016345281153917313 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38425925374031067 +jet/accuracy_1_of_1‚0.18998968601226807 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.503870964050293 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11644027382135391 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.7443277835845947 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3937770426273346 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897609949111938 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9703754186630249 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_2_of_2‚0.08214525133371353 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9899480938911438 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16084492206573486 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6198035478591919 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5294621586799622 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997110903263092 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9924427270889282 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8249976634979248 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16062448918819427 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8185693025588989 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.32319432497024536 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6301209330558777 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12200962007045746 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009380762465298176 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/specificity‚0.9055516123771667 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1213274896144867 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997110903263092 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841269254684448 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.254741907119751 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.982629120349884 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6375472545623779 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15860451757907867 *jet/accuracy_1_of_2‚0.5898876786231995 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11848651617765427 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35309261083602905 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1948234885931015 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.35229483246803284 _runtime‚ 697.490745979 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9875130653381348 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09928013384342194 trainer/global_step‚799 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4058458209037781 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10777004808187485 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.04620279371738434 +particle/sensitivity‚0.5304865837097168 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967706799507141 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.551360547542572 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.46869635581970215 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15923121571540833 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10911521315574646 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4452663064002991 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09794919937849045 particle/accuracy_1_of_3‚1¦ 4“’àõ{ :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04345620423555374 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9851877093315125 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2604166865348816 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.018708787858486176 +€i<9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8274613618850708 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.40007323026657104 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12000000476837158 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.05185672640800476 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5876806378364563 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.471001535654068 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4063183069229126 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961584806442261 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15829935669898987 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6537302136421204 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42400041222572327 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35185185074806213 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972054362297058 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4119214117527008 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5208333730697632 particle/accuracy_2_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.982336699962616 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 epoch‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.42518553137779236 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_4‚1 jet/accuracy_4_of_5‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.30558475852012634 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.996457576751709 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940527677536011 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.36433401703834534 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4631331264972687 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5103174448013306 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924328923225403 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.04631427675485611 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.43367084860801697 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017240718007087708 .particle/accuracy_3_of_4‚0.989321768283844 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.031395573168992996 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0362953320145607 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.03185763955116272 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.47587522864341736 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0012210012646391988 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.42643070220947266 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.38200193643569946 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06552673131227493 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.048396866768598557 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.389187753200531 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/f_score‚0.5655713081359863 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _step‚65 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9951465129852295 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47513702511787415 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.38509315252304077 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010342216119170189 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.7673367857933044 ,jet/accuracy_4_of_4‚0.007130124606192112 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3823491632938385 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5115395784378052 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4383530914783478 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09518452733755112 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3671875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.02506919763982296 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.029035761952400208 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931998252868652 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895977973937988 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4475761950016022 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.016345281153917313 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38425925374031067 +jet/accuracy_1_of_1‚0.18998968601226807 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.503870964050293 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11644027382135391 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.7443277835845947 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3937770426273346 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897609949111938 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9703754186630249 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_2_of_2‚0.08214525133371353 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9899480938911438 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16084492206573486 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6198035478591919 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5294621586799622 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.997110903263092 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9924427270889282 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8249976634979248 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16062448918819427 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8185693025588989 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.32319432497024536 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6301209330558777 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12200962007045746 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009380762465298176 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/specificity‚0.9055516123771667 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1213274896144867 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.997110903263092 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841269254684448 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.254741907119751 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.982629120349884 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6375472545623779 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15860451757907867 *jet/accuracy_1_of_2‚0.5898876786231995 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11848651617765427 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35309261083602905 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1948234885931015 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.35229483246803284 _runtime‚ 697.490745979 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9875130653381348 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09928013384342194 trainer/global_step‚799 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4058458209037781 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10777004808187485 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.04620279371738434 +particle/sensitivity‚0.5304865837097168 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967706799507141 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.551360547542572 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.46869635581970215 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15923121571540833 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10911521315574646 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4452663064002991 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09794919937849045 particle/accuracy_1_of_3‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9003491997718811 7validation_average_jet_accuracy‚0.23225554823875427 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5086235404014587 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917953014373779 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5213876962661743 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06075487285852432 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.26980650424957275 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.42468637228012085 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.10138918459415436 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.44300320744514465 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3533293604850769 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47835230827331543 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006349206669256091 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1610439121723175 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *loss/val_total_loss‚15.525659529837057 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9931291937828064 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 /particle/accuracy_3_of_3‚0.9990123510360718 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04264017567038536 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_4‚0.34754660725593567 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9074073433876038 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4418976604938507 9 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.058150313794612885 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5073750615119934 particle/accuracy_1_of_2‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005248802248388529 % _timestamp‚1.7842735053643408e+09 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.797224223613739 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08172374963760376 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6064608693122864 particle/accuracy_1_of_1‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.01918800175189972 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4844214916229248 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.01949610747396946 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4418976604938507 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.01949610747396946 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.01549927145242691 +jet/accuracy_2_of_3‚0.27307969331741333 particle/accuracy_2_of_2‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8536737561225891 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2568112909793854 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6198035478591919 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3646792769432068 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9131760597229004 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.35229483246803284 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.45666664838790894 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15829935669898987 jet/accuracy_2_of_5‚1A)¹†Ÿá‚(:” è´çÒŒ®¥ gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÂâ‚(:ù è´çÒöÛɧ disk.rbd0.in‚0$ memory_percent‚96.96347713470459 network.sent‚ 1.17776e+06 disk.rbd1.out‚1.796875! disk.dm-0.out‚1141.0830078125 disk.nvme0n1.in‚ 33.55859375& disk.nvme0c0n1.out‚1141.0830078125 disk.sda2.in‚0 disk.dm-1.out‚19.96875( proc.memory.availableMB‚ 497.50390625$ disk./.usageGB‚548.7025299072266 disk.drbd2165.in‚0 disk.sda3.out‚19.96875 disk.rbd2.out‚0" proc.memory.rssMB‚ 2743.6484375* proc.memory.percent‚16.745901107788086$ disk.nvme0n1.out‚1141.0830078125 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 network.recv‚233019 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 33.55859375* disk./.usagePercent‚36.823887174161804! disk.nvme0c0n1.in‚ 33.55859375 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚19.96875 cpu‚25.3984348382301 proc.cpu.threads‚172æŽBžã‚(:“ ÷´çÒ€†æ§ gpu.0.gpu‚18.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡¸œÄä‚(:÷ ÷´çÒß×ý¬* disk./.usagePercent‚36.823617598228246 disk.rbd3.in‚0" disk.dm-0.out‚1396.17431640625 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.222221e+06 disk.rbd1.in‚0% disk.nvme0n1.out‚1396.17431640625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1396.17431640625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 34.38671875 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 34.38671875 disk.sda3.out‚ 21.0703125 network.recv‚237539 disk.rbd3.out‚0 proc.memory.rssMB‚ 2744.46875 disk.rbd0.in‚0 disk.sda.out‚ 21.0703125 disk.sda2.out‚0 disk.dm-1.out‚ 21.0703125) proc.memory.percent‚16.75090789794922 disk.rbd1.out‚1.875 disk.drbd2165.in‚0! proc.memory.availableMB‚497.5 cpu‚25.412785121388072$ disk./.usageGB‚548.6985130310059 disk.rbd2.out‚0 disk.nvme0n1.in‚ 34.38671875 disk.sda1.in‚0# memory_percent‚96.9635009765625¸}žå‚(:“ †µçҵد¤ gpu.0.gpu‚67.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚40.0 gpu.0.powerWatts‚236.479' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚47.29580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Öüæ‚(:ñ †µçÒ·¹½¦) proc.memory.percent‚16.75090789794922 network.sent‚ 1.227294e+06) disk./.usagePercent‚36.82377657890701 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 cpu‚25.42771663798256 disk.nvme0n1.in‚ 35.1015625 disk.nvme0c0n1.in‚ 35.1015625# disk.nvme0c0n1.out‚ 1412.9140625 disk.sda3.out‚ 23.12890625 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0! disk.nvme0n1.out‚ 1412.9140625 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-0.out‚ 1412.9140625 disk.rbd1.in‚0$ memory_percent‚96.96652889251709 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 35.1015625( proc.memory.availableMB‚ 497.00390625$ disk./.usageGB‚548.7008819580078 disk.dm-1.out‚ 23.12890625 proc.memory.rssMB‚ 2744.46875 proc.cpu.threads‚17 network.recv‚240898 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 23.12890625 disk.rbd1.out‚1.875sÐ,C0ç+ _wandbruntime‚740  _runtime‚7404`OÚÆèÀ ' lr-AdamW/pg1‚0.00047205644243682247 ' lr-AdamW/pg2‚0.00047205644243682247 trainer/global_step‚849 % _timestamp‚1.7842735482028856e+09 _runtime‚ 740.328547628 _step‚66{åžÊéÄ trainer/global_step‚849 % _timestamp‚1.7842735482028856e+09 _runtime‚ 740.328547628 _step‚66 ' lr-AdamW/pg1‚0.00047205644243682247 ' lr-AdamW/pg2‚0.00047205644243682247Bi§ê©0ê+ _wandbruntime‚740  _runtime‚740¯®&Æ'ëÀO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3329418 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129283532254185 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7873091 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29478818041344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 908666919936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129049849470976 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 896510368491 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129947713351680 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 898222258688 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3329689 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129948071729664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129948071729664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7863115 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129947713351680 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1494168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129049832431616 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6732392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129049849470976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29478914745344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7863115 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129947713351680 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_tra,2tî!in_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1494168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129049832431616 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6732392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129049849470976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29478914745344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29478818041344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 908666919936 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129049849470976 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 896510368491 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129947713351680 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 898222258688 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3329689 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129948071729664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129948071729664C‘ôl0í+ _wandbruntime‚740  _runtime‚740MèÙÑ&îËM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 trainer/global_step‚849 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚130096325393408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129197090500608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129197073461248 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6740024 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3333228 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128533192007680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28603676583936 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 909604403200 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7882157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129197090500608 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 899234892800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899234892800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7872167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 740.945565856 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6740026 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7872167 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1495913 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1837584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1837315 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3333500 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29513262374912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128533174968320 _step‚68 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898893547008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29513369557504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898893547008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14612191 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7882157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129197090500608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129197073461248 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6740024 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3333228 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128533192007680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28603676583936 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 909604403200 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7882157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129197090500608 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 899234892800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899234892800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7872167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 740.945565856 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6740026 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7872167 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1495913 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1837584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1837315 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3333500 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29513262374912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128533174968320 _step‚68 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 898893547008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29513369557504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898893547008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14612191 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7882157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474DO§0ð+ _wandbruntime‚740  _runtime‚740Ž.}ñý 1loss/FBt2/assignment_loss‚0.16096279340272304 /loss/FBt1/detection_loss‚1.1039136140935324 trainer/global_step‚849 /loss/FRt2/detection_loss‚1.4977895466217261 % _timestamp‚1.7842735488213263e+09 1loss/SRqqt1/assignment_loss‚1.317206978772397 1loss/FBt1/assignment_loss‚0.12066773987802623 /loss/FRt1/detection_loss‚0.7141390610854179 1loss/SRqqt1/detection_loss‚1.4503886493509763 1loss/SRqqt2/detection_loss‚0.7797063586767763 epoch‚2 .loss/FBt2/detection_loss‚1.733256495697424 &loss/total_loss‚15.263006815393275 _runtime‚ 740.946707401 _step‚69 0loss/FRt1/assignment_loss‚3.2310381396281542 0loss/FRt2/assignment_loss‚2.2528506979064153 2loss/SRqqt2/assignment_loss‚0.9010867402797046Üh™¡ò 0loss/FRt1/assignment_loss‚3.2310381396281542 0loss/FRt2/assignment_loss‚2.2528506979064153 2loss/SRqqt2/assignment_loss‚0.9010867402797046 1loss/FBt2/assignment_loss‚0.16096279340272304 /loss/FBt1/detection_loss‚1.1039136140935324 trainer/global_step‚849 /loss/FRt2/detection_loss‚1.4977895466217261 % _timestamp‚1.7842735488213263e+09 1loss/SRqqt1/assignment_loss‚1.317206978772397 1loss/FBt1/assignment_loss‚0.12066773987802623 /loss/FRt1/detection_loss‚0.7141390610854179 1loss/SRqqt1/detection_loss‚1.4503886493509763 1loss/SRqqt2/detection_loss‚0.7797063586767763 epoch‚2 .loss/FBt2/detection_loss‚1.733256495697424 &loss/total_loss‚15.263006815393275 _runtime‚ 740.946707401 _step‚69EÙ›ð_žó‚(:“ •µçÒÖí£ gpu.0.gpu‚88.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚40.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ì0Þô‚(:õ •µçÒ‡˜¦$ disk./.usageGB‚552.3785285949707 disk.rbd2.in‚0# disk.nvme0n1.out‚5189.744140625 proc.memory.rssMB‚ 2744.46875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.out‚0 disk.sda.out‚ 23.6328125 disk.rbd3.in‚0 disk.dm-0.in‚ 35.9296875 disk.rbd3.out‚0 disk.dm-0.out‚5189.744140625 cpu‚25.437269203727286 network.sent‚ 1.24536e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚37.07058653046465 disk.nvme0n1.in‚ 35.9296875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 35.9296875% disk.nvme0c0n1.out‚5189.744140625 disk.sda2.in‚0 disk.sda3.out‚ 23.6328125 disk.dm-1.out‚ 23.6328125$ memory_percent‚96.96505069732666) proc.memory.percent‚16.75090789794922 disk.rbd1.out‚1.9375 disk.rbd0.out‚0 network.recv‚244454 disk.sda1.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 497.24609375 ?9žõ‚(:“ ¤µçÒ²ó¤¤ gpu.0.gpu‚98.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚40.0 gpu.0.powerWatts‚279.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.867999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:¯9ö‚(:ý ¤µçÒµÍÀ¦ disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚0 disk.nvme0c0n1.in‚ 35.9296875 disk.dm-1.out‚ 23.89453125 network.recv‚248671 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚5193.07568359375 disk.sda.out‚ 23.89453125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚1.9375" disk.dm-0.out‚5193.07568359375 network.sent‚ 1.297348e+06) disk./.usagePercent‚37.07061469129342$ disk./.usageGB‚552.3789482116699 disk.rbd0.out‚0) proc.memory.percent‚16.75090789794922 cpu‚25.45383148954805 proc.cpu.threads‚17 disk.rbd2.in‚0% disk.nvme0n1.out‚5193.07568359375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 35.9296875 disk.sda3.in‚0' proc.memory.availableMB‚ 497.1484375 disk.nvme0n1.in‚ 35.9296875 disk.sda1.out‚0 disk.sda3.out‚ 23.89453125 disk.sda1.in‚0$ memory_percent‚96.96564674377441 proc.memory.rssMB‚ 2744.46875ŒâÐ7÷‚(:‰ ³µçÒ»þƒ¥ gpu.0.gpu‚88.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚40.0 gpu.0.powerWatts‚262.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5178 gpu.0.smClock‚1410 gpu*KÁ`].0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v§ø‚(:‡ ³µçҫ̳§ network.recv‚252123" disk.dm-0.out‚5201.22900390625 disk.sda.in‚0 disk.sda3.in‚0 disk.sda.out‚ 24.20703125( proc.memory.availableMB‚ 497.88671875$ disk./.usageGB‚552.3799133300781 disk.rbd1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.745877265930176) disk./.usagePercent‚37.07067946119958 disk.sda2.out‚0 disk.dm-0.in‚ 36.64453125 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 36.64453125 disk.dm-1.out‚ 24.20703125 disk.rbd1.out‚2.015625 disk.nvme0n1.in‚ 36.64453125 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚0% disk.nvme0n1.out‚5201.22900390625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0# memory_percent‚96.9611406326294 disk.rbd3.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚5201.22900390625# proc.memory.rssMB‚ 2743.64453125 cpu‚25.469450510618366 disk.sda3.out‚ 24.20703125 disk.rbd3.in‚0 network.sent‚ 1.302485e+06úÑ"ì0ù+ _wandbruntime‚781  _runtime‚781tÁOÄú¾ % _timestamp‚1.7842735888987794e+09 _runtime‚ 781.025528815 _step‚70 & lr-AdamW/pg1‚0.0004719126807037754 & lr-AdamW/pg2‚0.0004719126807037754 trainer/global_step‚899åÁæÈû trainer/global_step‚899 % _timestamp‚1.7842735888987794e+09 _runtime‚ 781.025528815 _step‚70 & lr-AdamW/pg1‚0.0004719126807037754 & lr-AdamW/pg2‚0.0004719126807037754FJñÎû0ü+ _wandbruntime‚781  _runtime‚781zçïÆ'ýÀO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30241685751808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 % _timestamp‚1.7842735889025254e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8317567 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137360818713088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 948576421888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959946567680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1581421 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136411883913216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1581418 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 960026756608 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3520726 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31201613707264 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 947467405285 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 948576421888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚136659682334693 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135712197890048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 947126855399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8317567 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8327543 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1939563 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1939308 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚899 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7113992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30241667139584 _step‚71 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3520984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137360460335104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 948917760512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚136411883913216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136659324745447 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135712214929408 _runtime‚ 781.028376962 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136411900952576 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 948917760512 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7113994 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Ü"ßÊ'þÄO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30241685751808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 % _timestamp‚1.7842735889025254e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8317567 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137360818713088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 948576421888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959946567680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1581421 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136411883913216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1581418 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 960026756608 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3520726 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31201613707264 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 947467405285 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 948576421888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚136659682334693 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135712197890048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 947126855399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8317567 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8327543 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1939563 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1939308 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚899 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7113992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30241667139584 _step‚71 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3520984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137360460335104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 948917760512 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚136411883913216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136659324745447 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135712214929408 _runtime‚ 781.028376962 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136411900952576 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 948917760512 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7113994 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0GåìE@0ÿ+ _wandbruntime‚781  _runtime‚781¼|Ñ&€ËM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31236070361600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135858708561920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137509072376832 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136559141982208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136807204060399 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121624 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 949589048832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30275095898112 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 948478459119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1583166 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1941594 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _step‚72 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 949930394624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 949930394624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137508713991680 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15448243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 948137909121 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8326619 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30275114510336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚899 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7121626 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136559124942848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1583163 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136806846471041 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 961056742400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100891136 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 960974463488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3524498 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135858725601280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358ÇÜŸÇÏM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15448243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 948137909121 MADeviceStatsMonitor.on_train_batch_endÿKañ$/allocation.small_pool.freed‚8326619 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30275114510336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚899 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7121626 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136559124942848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1583163 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136806846471041 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 961056742400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100891136 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 960974463488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3524498 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135858725601280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31236070361600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135858708561920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137509072376832 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136559141982208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136807204060399 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121624 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 949589048832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30275095898112 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 948478459119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1583166 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1941594 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _step‚72 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 949930394624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 949930394624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137508713991680 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144773207318528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143773952434176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144035474825955 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2043142 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1011508571136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143773952434176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8781995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2043414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 999613262336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143773935394816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842736292688038e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7495592 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495594 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31913105065984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 997743342307 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 999271923712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3712085 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143037731483648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144773565696512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8781995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144773565696512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3711810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 999271923712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _runtime‚ 821.398383549 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8772019 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8772019 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 _step‚75 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 999613262336 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2043414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 999613262336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143773935394816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842736292688038e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7495592 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495594 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31913105065984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 997743342307 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 999271923712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3712085 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143037731483648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144773565696512 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8781995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144773565696512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3711810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 999271923712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _runtime‚ 821.398383549 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8772019 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8772019 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 _step‚75 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 999613262336 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144773207318528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143773952434176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144035474825955 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2043142 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1011508571136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143773952434176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616K©˜0+ _wandbruntime‚822  _runtime‚822jËÓ$Ö&ÐM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144182996551549 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143184259194880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31946533824512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1012543371776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143921193463808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143921176424448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105085440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1000284550656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2045437 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 _step‚76 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 998754396029 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32959095808512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7503224 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3715593 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 $ _timestamp‚1.784273629883671e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144921819360256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1000284550656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1000625896448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1670413 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86473216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚ 822.009488302 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2045180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16294287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7503226 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1670416 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8791061 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144183354140907 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31946552436736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 trainer/global_step‚949 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143921176424448 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7503226 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32958990723072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1012456898560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999094946027 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999094946027 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143921193463808 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143921176424448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105085440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1000284550656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2045437 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 _step‚76 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 998754396029 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32959095808512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7503224 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3715593 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 $ _timestamp‚1.784273629883671e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144921819360256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1000284550656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1000625896448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1670413 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86473216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚ 822.009488302 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2045180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16294287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7503226 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1670416 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8791061 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144183354140907 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31946552436736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 trainer/global_step‚949 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143921176424448 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7503226 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32958990723072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1012456898560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151411624906463 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151135986876416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1050308764160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062939244544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842736698288205e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1062859055616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1049967425536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9236447 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1755918 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7877192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151136003915776 d´ÊýuQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33584561604608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7877192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1050308764160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1755921 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2147259 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34647402048000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 861.958398456 _step‚79 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7877194 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9236447 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33584542992384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152186312679936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152185954301952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1048700379101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3903177 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169ÇôD+Ò' ÌO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9226471 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2147539 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152185954301952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1048700379101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3903177 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34647500849152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151411982495709 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150363282116608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1048359829215 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151136003915776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151135986876416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151411624906463 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151135986876416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1050308764160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062939244544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842736698288205e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1062859055616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1049967425536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9236447 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1755918 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7877192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151136003915776 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33584561604608 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7877192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1050308764160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1755921 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2147259 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34647402048000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 861.958398456 _step‚79 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7877194 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9236447 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33584542992384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152186312679936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152334207958528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884826 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34681859906560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1757666 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1049370882937 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9235523 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150509792788480 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151283244945408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7884826 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7884824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1063888155648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1063968337408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3907001 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33617990363136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1051321398272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚80 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884824 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34681958700544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150509775749120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152334566343680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 % _timestamp‚1.7842736704425664e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152334566343680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151283244945408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1050980052480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 862.568460454j>‘yŽ£ÖM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3907001 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33617990363136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1051321398272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚80 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884824 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34681958700544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150509775749120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152334566343680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 % _timestamp‚1.7842736704425664e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152334566343680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151283244945408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1050980052480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 862.568460454 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151283227906048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 trainer/global_step‚999 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152334207958528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7884826 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34681859906560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1757666 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1049370882937 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9235523 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150509792788480 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151283244945408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7884826 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7884824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1063888155648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1063968337408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268PÛIó&0¤+ _wandbruntime‚862  _runtime‚862Ñ¢Q¥ü /loss/FRt1/detection_loss‚0.6268454829354926 /loss/FBt1/detection_loss‚0.9172969123529071 epoch‚2 trainer/global_step‚999 % _timestamp‚1.7842736704442325e+09 _runtime‚ 862.569601929 1loss/SRqqt1/detection_loss‚1.3271698228142284 _step‚81 2loss/SRqqt2/assignment_loss‚0.8221075009900407 .loss/FRt2/detection_loss‚1.472115282022838 1loss/SRqqt2/detection_loss‚0.8326346529057348 &loss/total_loss‚13.985197406040106 2loss/SRqqt1/assignment_loss‚1.0387402524802465 1loss/FBt1/assignment_loss‚0.13919674054731124 0loss/FBt2/assignment_loss‚0.1207196012129185 /loss/FBt2/detection_loss‚1.5854350044999435 /loss/FRt1/assignment_loss‚3.245644206488515 0loss/FRt2/assignment_loss‚1.8572919467899287ªh%v¦€ 2loss/SRqqt2/assignment_loss‚0.8221075009900407 .loss/FRt2/detection_loss‚1.472115282022838 1loss/SRqqt2/detection_loss‚0.8326346529057348 &loss/total_loss‚13.985197406040106 2loss/SRqqt1/assignment_loss‚1.0387402524802465 1loss/FBt1/assignment_loss‚0.13919674054731124 0loss/FBt2/assignment_loss‚0.1207196012129185 /loss/FBt2/detection_loss‚1.5854350044999435 /loss/FRt1/assignment_loss‚3.245644206488515 0loss/FRt2/assignment_loss‚1.8572919467899287 /loss/FRt1/detection_loss‚0.6268454829354926 /loss/FBt1/detection_loss‚0.9172969123529071 epoch‚2 trainer/global_step‚999 % _timestamp‚1.7842736704442325e+09 _runtime‚ 862.569601929 1loss/SRqqt1/detection_loss‚1.3271698228142284 _step‚81Q×Q?3§‚(:’ ¶çÒÙÄ¥ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.6' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.919999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õB;ƒ¨‚(:ú ¶çÒ€É⧠proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 40.55859375 network.sent‚ 1.445716e+06 disk.rbd1.in‚0 disk.dm-0.out‚5287.365234375$ memory_percent‚96.96540832519531$ proc.memory.availableMB‚497.1875# proc.memory.rssMB‚ 2744.47265625# disk./.usageGB‚549.267219543457 disk.dm-0.in‚ 40.55859375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 26.46484375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 26.46484375 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 26.46484375 disk.rbd1.out‚2.28125 disk.rbd0.in‚0 cpu‚25.5568081870465% disk.nvme0c0n1.out‚5287.365234375 disk.rbd3.out‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 40.55859375# disk.nvme0n1.out‚5287.365234375 disk.rbd0.out‚0 disk.sda.in‚0) proc.memory.percent‚16.75093173980713 disk.dm-1.in‚0* disk./.usagePercent‚36.861783969455395 disk.rbd3.in‚0 disk.rbd2.in‚0 network.recv‚274093WÄ·7ž©‚(:“ œ¶çÒù€„¤ gpu.0.gpu‚45.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.184' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.63680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#»¥Ý ª‚(:ÿ œ¶çÒ¹¬Š¦ cpu‚25.569690963323723 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 41.2734375 disk.rbd3.in‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚5300.2587890625 disk.sda.out‚ 26.69921875 disk.rbd1.out‚2.28125# memory_percent‚98.4684705734253) proc.memory.percent‚16.75093173980713) disk./.usagePercent‚36.86174915243073 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚5300.2587890625 network.recv‚278563 disk.rbd0.in‚0 disk.rbd2.out‚0 network.sent‚ 1.510731e+06 disk.sda3.in‚0 disk.sda3.out‚ 26.69921875 disk.sda1.out‚0( proc.memory.availableMB‚ 250.92578125 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 26.69921875# proc.memory.rssMB‚ 2744.47265625 disk.rbd3.out‚0! disk.dm-0.out‚5300.2587890625 disk.nvme0n1.in‚ 41.2734375 disk.sda1.in‚0$ disk./.usageGB‚549.2667007446289 disk.dm-0.in‚ 41.2734375C,’”«‚(:‰ «¶çÒ¼×ë¤ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’aÅ(:„ «¶çÒ±‡Â§ disk.rbd1.out‚2.359375$ memory_percent‚98.46842288970947# proc.memory.rssMB‚ 2744.47265625 disk.rbd2.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.582781937072998 disk.rbd2.in‚0' disk.nvme0c0n1.out‚5303.70263671875 disk.rbd3.out‚0 network.recv‚282243) disk./.usagePercent‚36.86173353597115 disk.nvme0n1.in‚ 41.2734375 disk.nvme0c0n1.in‚ 41.2734375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 250.93359375% disk.nvme0n1.out‚5303.70263671875 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚5303.70263671875$ disk./.usageGB‚549.2664680480957 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 41.2734375) proc.memory.percent‚16.75093173980713 network.sent‚ 1.515912e+06 disk.sda.in‚0 disk.sda.out‚ 27.29296875 disk.sda3.out‚ 27.29296875 disk.dm-1.out‚ 27.29296875îÆÁþ0­+ _wandbruntime‚902  _runtime‚902ÿ…º«Å®¿ _step‚82 & lr-AdamW/pg1‚0.0004713906526678131 & lr-AdamW/pg2‚0.0004713906526678131 trainer/global_step‚1049 % _timestamp‚1.7842737104097314e+09 _runtime‚ 902.535412558&‘«ƒÉ¯Ã _runtime‚ 902.535412558 _step‚82 & lr-AdamW/pg1‚0.0004713906526678131 & lr-AdamW/pg2‚0.0004713906526678131 trainer/global_step‚1049 % _timestamp‚1.7842737104097314e+09Rá"50°+ _wandbruntime‚902  _runtime‚902#4 Ï'±ÉO OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9690899 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8258794 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158498038358016 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4094498 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159599059663360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9690899 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157688798670848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2251327 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1101004265984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159598701285376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8258792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159598701285376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101004265984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8258794 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1114350887424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚83 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158498055397376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1843168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158787774986971 $ _timestamp‚1.784273710410828e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2251059 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102995456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157688815710208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9680923 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1114266504192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 trainer/global_step‚1049 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158498038358016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1100662927360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35255980918784Oè™Ó'²ÍO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159598701285376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8258792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159598701285376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101004265984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8258794 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1114350887424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚83 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158498055397376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1843168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158787774986971 $ _timestamp‚1.784273710410828e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2251059 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102995456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157688815710208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9680923 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1114266504192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 trainer/global_step‚1049 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158498038358016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1100662927360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35255980918784 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9690899 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8258794 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158498038358016 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4094498 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159599059663360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9690899 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157688798670848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2251327 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1101004265984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9699965 FãîóŠbatch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157835326382080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17956399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159747313327104 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9699965 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36404715238912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1102016900096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1115383646208 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158645279387648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36404811935744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159747313327104 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9689975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100327919843 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4098336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17956399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158935296712565 % _timestamp‚1.7842737110230834e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158935654301923 trainer/global_step‚1049 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8266426 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35289409677312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157835309342720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1101675554304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 903.148923436 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1101675554304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4098056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1115305561600 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159746954941952 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9689975 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1102016900096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚84VúÅ’Ý&µ×M ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚84 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8266426 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1099987369845 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35289428289536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9699965 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17956399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159747313327104 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9699965 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36404715238912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1102016900096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1115383646208 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158645279387648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36404811935744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159747313327104 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9689975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100327919843 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4098336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17956399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158935296712565 % _timestamp‚1.7842737110230834e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚158935654301923 trainer/global_step‚1049 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8266426 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35289409677312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157835309342720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1101675554304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 _runtime‚ 903.148923436 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1101675554304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4098056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1115305561600 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159746954941952 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9689975 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1102016900096T0• 90¶+ _wandbruntime‚903  _runtime‚903Ïêq ·ý 2loss/SRqqt1/assignment_loss‚1.1272908424623826 2loss/SRqqt2/assignment_loss‚0.7819748072166478 .loss/FBt1/detection_loss‚0.816443507219227 % _timestamp‚1.7842737110253985e+09 _runtime‚ 903.150744725 /loss/FRt1/detection_loss‚0.6078542719557202 /loss/FBt2/detection_loss‚1.5381602622568606 trainer/global_step‚1049 0loss/FBt1/assignment_loss‚0.1158383221646157 2loss/FBt2/assignment_loss‚0.029194834686268222 &loss/total_loss‚13.599392903583158 _step‚85 /loss/FRt1/assignment_loss‚3.116237476635515 /loss/FRt2/assignment_loss‚1.643279300638265 /loss/FRt2/detection_loss‚1.5164877051267078 1loss/SRqqt1/detection_loss‚1.5206649860341428 1loss/SRqqt2/detection_loss‚0.7859665871868061 epoch‚2‰Sȸ 2loss/SRqqt1/assignment_loss‚1.1272908424623826 2loss/SRqqt2/assignment_loss‚0.7819748072166478 .loss/FBt1/detection_loss‚0.816443507219227 % _timestamp‚1.7842737110253985e+09 _runtime‚ 903.150744725 /loss/FRt1/detection_loss‚0.6078542719557202 /loss/FBt2/detection_loss‚1.5381602622568606 trainer/global_step‚1049 0loss/FBt1/assignment_loss‚0.1158383221646157 2loss/FBt2/assignment_loss‚0.029194834686268222 &loss/total_loss‚13.599392903583158 _step‚85 /loss/FRt1/assignment_loss‚3.116237476635515 /loss/FRt2/assignment_loss‚1.643279300638265 /loss/FRt2/detection_loss‚1.5164877051267078 1loss/SRqqt1/detection_loss‚1.5206649860341428 1loss/SRqqt2/detection_loss‚0.7859665871868061 epoch‚2U"iy^ž¹‚(:“ º¶çÒˆӤ gpu.0.gpu‚95.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xß`6 º‚(:ÿ º¶çÒ·ìø¦ cpu‚25.594274084795526 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚285352 disk.dm-1.out‚ 27.5546875 disk.rbd2.out‚0( proc.memory.availableMB‚ 250.72265625 network.sent‚ 1.531875e+06* disk./.usagePercent‚36.861750944483475 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚98.46971035003662# proc.memory.rssMB‚ 2744.47265625 disk.dm-1.in‚0 disk.sda3.out‚ 27.5546875! disk.dm-0.out‚5313.8974609375) proc.memory.percent‚16.75093173980713$ disk./.usageGB‚549.2667274475098 disk.rbd0.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚5313.8974609375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 42.1015625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 42.1015625& disk.nvme0c0n1.out‚5313.8974609375 disk.sda.out‚ 27.5546875 disk.rbd1.out‚2.359375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 42.1015625ï¸EŸ»‚(:” ɶçÒÒé’¥ gpu.0.gpu‚97.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†f ¼‚(:€ ɶçÒöȧ disk.dm-0.in‚ 42.81640625! disk.dm-0.out‚5322.2978515625 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚98.46782684326172 disk.sda2.in‚0 network.recv‚290109# disk./.usageGB‚549.268726348877 disk.rbd2.out‚0 disk.rbd3.in‚0 cpu‚25.606628902116256 network.sent‚ 1.58387e+06* disk./.usagePercent‚36.861885092431415 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 42.81640625& disk.nvme0c0n1.out‚5322.2978515625 disk.sda3.out‚ 27.82421875 disk.dm-1.out‚ 27.82421875 disk.rbd1.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2744.47265625 disk.sda1.out‚0% proc.memory.availableMB‚ 251.03125) proc.memory.percent‚16.75093173980713 disk.rbd1.out‚2.421875 disk.drbd2165.in‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚5322.2978515625! disk.nvme0c0n1.in‚ 42.81640625 disk.sda.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚ 27.82421875 disk.dm-1.in‚0¦Aeõ0½+ _wandbruntime‚942  _runtime‚942ĉkmƾÀ _step‚86 ' lr-AdamW/pg1‚0.00047118643296461545 ' lr-AdamW/pg2‚0.00047118643296461545 trainer/global_step‚1099 $ _timestamp‚1.784273750752741e+09 _runtime‚ 942.878386837ó ïÊ¿Ä $ _timestamp‚1.784273750752741e+09 _runtime‚ 942.878386837 _step‚86 ' lr-AdamW/pg1‚0.00047118643296461545 ' lr-AdamW/pg2‚0.00047118643296461545 trainer/global_step‚1099V×i[ø0À+ _wandbruntime‚942  _runtime‚942!M1Ô'ÁÎO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166164282656725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1149933352917 trainer/global_step‚1099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167011448268800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165014349303808 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640394 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167011806646784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167011806646784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38093078987264 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165860089839616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10145351 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285959 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1930421 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1930418 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2355541 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165860089839616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1151358429184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10135375 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1165757108224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1165660142080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1151699767808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2355825 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96966144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165014332264448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10135375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1149592803031 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 942.879652116¿}~—àÂÒO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 ]JDeviceStatsMonitor.on_train_batch_start/%ÆÜ ø&inactive_split_bytes.all.allocated‚38093194565632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4286246 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8640392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10145351 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165860089839616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10145351 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285959 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1930421 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1930418 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2355541 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165860089839616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1151358429184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10135375 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1165757108224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1165660142080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1151699767808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2355825 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96966144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165014332264448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10135375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1149592803031 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 942.879652116 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166164282656725 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1149933352917 trainer/global_step‚1099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167011448268800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165014349303808 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640394 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167011806646784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167011806646784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38093078987264 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4289775 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1150944406751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10154417 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚88 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1152712401920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4290044 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1932163 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165160859975680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚1099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166007330869248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1932166 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2357878 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1152371056128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167159701925376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38127640612352 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10144427 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36960866215936 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8648024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8648026 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38127533429760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165160842936320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872Øá!3â&ÅÜM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1932163 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165160859975680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚1099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166007330869248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1932166 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2357878 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1152371056128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167159701925376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38127640612352 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10144427 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36960866215936 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8648024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8648026 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38127533429760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165160842936320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1150603856753 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4289775 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1150944406751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10154417 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚88 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1152712401920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4290044 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0XäÙô0Æ+ _wandbruntime‚943  _runtime‚9439&FÇú 0loss/FRt1/assignment_loss‚3.1351990529269433 .loss/FRt1/detection_loss‚0.618256163766107 .loss/FRt2/detection_loss‚1.495870546654939 &loss/total_loss‚14.133885138073234 _step‚89 1loss/SRqqt2/detection_loss‚0.8165552995946092 /loss/FBt1/detection_loss‚0.7918500909529947 /loss/FBt2/detection_loss‚1.7538641597265783 % _timestamp‚1.7842737513676019e+09 /loss/FRt2/assignment_loss‚1.731156716364112 1loss/FBt1/assignment_loss‚0.15215612980411852 epoch‚2 _runtime‚ 943.493349968 2loss/SRqqt1/assignment_loss‚1.0546919989089172 0loss/SRqqt2/assignment_loss‚0.81543340853618 1loss/FBt2/assignment_loss‚0.32277305895568925 0loss/SRqqt1/detection_loss‚1.446078511882046 trainer/global_step‚1099éÂïÈþ 1loss/SRqqt2/detection_loss‚0.8165552995946092 /loss/FBt1/detection_loss‚0.7918500909529947 /loss/FBt2/detection_loss‚1.7538641597265783 % _timestamp‚1.7842737513676019e+09 /loss/FRt2/assignment_loss‚1.731156716364112 1loss/FBt1/assignment_loss‚0.15215612980411852 epoch‚2 _runtime‚ 943.493349968 2loss/SRqqt1/assignment_loss‚1.0546919989089172 0loss/SRqqt2/assignment_loss‚0.81543340853618 1loss/FBt2/assignment_loss‚0.32277305895568925 0loss/SRqqt1/detection_loss‚1.446078511882046 trainer/global_step‚1099 0loss/FRt1/assignment_loss‚3.1351990529269433 .loss/FRt1/detection_loss‚0.618256163766107 .loss/FRt2/detection_loss‚1.495870546654939 &loss/total_loss‚14.133885138073234 _step‚89Y uîM”É‚(:‰ ضçÒ­âö¤ gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡6 Ê‚(:‚ ضçÒÿϲ­ disk.drbd2165.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚5334.04736328125$ memory_percent‚98.47137928009033 disk.dm-1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 43.64453125 disk.sda.out‚ 28.1171875# proc.memory.rssMB‚ 2744.47265625 disk.sda2.out‚0 disk.rbd1.out‚2.421875 disk.sda1.in‚0 disk.sda.in‚0) proc.memory.percent‚16.75093173980713 cpu‚25.61756948173984 disk.sda3.out‚ 28.1171875 disk.drbd2165.in‚0 network.recv‚293223 disk.dm-1.out‚ 28.1171875 disk.dm-0.in‚ 43.64453125 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 43.64453125 disk.rbd0.out‚0 disk.rbd3.in‚0" disk.dm-0.out‚5334.04736328125 proc.cpu.threads‚17Ïüü network.sent‚ 1.586834e+06 disk.sda2.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 250.44921875( disk./.usagePercent‚36.8618789482506$ disk./.usageGB‚549.2686347961426 disk.rbd3.out‚0% disk.nvme0n1.out‚5334.04736328125zǔ˂(:‰ ç¶çÒ¦·¢£ gpu.0.gpu‚51.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&÷ƒHúÌ‚(:ï ç¶çÒ÷ôê¥ disk.nvme0c0n1.in‚ 44.359375% disk.nvme0c0n1.out‚5341.091796875 disk.sda3.out‚ 28.359375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 44.359375 disk.rbd0.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2744.47265625) proc.memory.percent‚16.75093173980713) disk./.usagePercent‚36.86188304437115 disk.sda2.out‚0 disk.dm-1.in‚0# disk.nvme0n1.out‚5341.091796875 disk.sda.out‚ 28.359375 disk.sda2.in‚0 disk.rbd2.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 network.recv‚297821 disk.dm-1.out‚ 28.359375$ memory_percent‚98.47159385681152' proc.memory.availableMB‚ 250.4140625 proc.cpu.threads‚17 disk.dm-0.out‚5341.091796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚2.5 network.sent‚ 1.651886e+06$ disk./.usageGB‚549.2686958312988 disk.rbd3.in‚0 disk.drbd2165.in‚0 cpu‚25.629645613463786 disk.rbd3.out‚0 disk.dm-0.in‚ 44.359375 disk.rbd2.in‚0 disk.rbd0.in‚0¼p-F“Í‚(:ˆ ö¶çҫʤ¥ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚282.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äñ#é΂(:ø ö¶çÒáìʧ disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 28.62890625 disk.rbd1.in‚0 disk.rbd3.out‚0 network.recv‚300884 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd1.out‚2.5' proc.memory.availableMB‚ 250.6015625 network.sent‚ 1.654808e+06 disk.rbd2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 28.62890625$ disk.nvme0n1.out‚5346.8837890625 disk.sda1.out‚0& disk.nvme0c0n1.out‚5346.8837890625$ memory_percent‚98.47044944763184) proc.memory.percent‚16.75093173980713 cpu‚25.639405742976972$ disk./.usageGB‚549.2686424255371 disk.nvme0c0n1.in‚ 44.359375 disk.sda.in‚0 disk.sda3.out‚ 28.62890625# proc.memory.rssMB‚ 2744.47265625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 44.359375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 44.359375) disk./.usagePercent‚36.86187946026566 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.dm-0.out‚5346.8837890625.Ê 0Ï+ _wandbruntime‚983  _runtime‚9839>s€ÆÐÀ ' lr-AdamW/pg1‚0.00047096712913537604 ' lr-AdamW/pg2‚0.00047096712913537604 trainer/global_step‚1149 % _timestamp‚1.7842737911406655e+09 _runtime‚ 983.26620829 _step‚9068KõÊÑÄ ' lr-AdamW/pg2‚0.00047096712913537604 trainer/global_step‚1149 % _timestamp‚1.7842737911406655e+09 _runtime‚ 983.26620829 _step‚90 ' lr-AdamW/pg1‚0.00047096712913537604Z!êÔ0Ò+ _wandbruntime‚983  _runtime‚9837ÿC™Ó'ÓÍO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9021992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2017671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174424553630208 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1202395269632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1217098020864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1200549839825 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174424195252224 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9021994 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1217007346176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172339865858048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚91 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38598856771584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1200209289939 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39815973404672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4477078 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1202053931008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2459152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174424553630208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9021994 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10589827 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172339882897408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1202053931008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2459152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174424553630208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9021994 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10589827 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172339882897408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174424195252224 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9021994 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1217007346176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172339865858048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚91[["ô0Õ+ _wandbruntime‚983  _runtime‚983¯Ñ»!Ö×M NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10598879 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚1149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10598879 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1218114211328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172486376529920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1218019349504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4480621 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173687954462939 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172486393569280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201560893659 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9029626 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173369382350848 _runtime‚ 983.880147806 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38632304142336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1201220343661 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173369399390208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39850304879616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9029626 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474048 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1203066557952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚92 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9029624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173369399390208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784273791754331e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38632285530112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2019413 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174572807293952 QEDeviceStatsMonitor.on_trai-Ám "n_batch_end/inactive_split.small_pool.freed‚2461208 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39850418353664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173369382350848 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4480897 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174572807293952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152ãŽØá&×ÛM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1218019349504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4480621 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173687954462939 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172486393569280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201560893659 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9029626 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173369382350848 _runtime‚ 983.880147806 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38632304142336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1201220343661 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173369399390208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39850304879616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9029626 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113474048 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1203066557952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚92 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9029624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173369399390208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784273791754331e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38632285530112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2019413 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174572807293952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2461208 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39850418353664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173369382350848 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4480897 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174572807293952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10598879 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚1149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10598879 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1218114211328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172486376529920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0\AU2û0Ø+ _wandbruntime‚983  _runtime‚983¼ºþWÙþ 1loss/FBt2/assignment_loss‚0.22979103183072563 /loss/FRt1/detection_loss‚0.6537611041833832 /loss/FBt1/detection_loss‚0.5047800939731615 /loss/FBt2/detection_loss‚1.1677893571279667 trainer/global_step‚1149 _runtime‚ 983.882224536 2loss/SRqqt1/assignment_loss‚1.1685575546519789 1loss/SRqqt2/detection_loss‚0.8483882227548921 0loss/FRt1/assignment_loss‚3.1421926282304993 0loss/FRt2/assignment_loss‚1.6990369341682656 .loss/FRt2/detection_loss‚1.360482978704465 &loss/total_loss‚13.044672716658118 _step‚93 2loss/SRqqt2/assignment_loss‚0.7851369997024521 0loss/FBt1/assignment_loss‚0.0968935976664041 1loss/SRqqt1/detection_loss‚1.3878622136639254 % _timestamp‚1.7842737917568786e+09 epoch‚2ÐÄ"Ú‚ /loss/FRt1/detection_loss‚0.6537611041833832 /loss/FBt1/detection_loss‚0.5047800939731615 /loss/FBt2/detection_loss‚1.1677893571279667 trainer/global_step‚1149 _runtime‚ 983.882224536 2loss/SRqqt1/assignment_loss‚1.1685575546519789 1loss/SRqqt2/detection_loss‚0.8483882227548921 0loss/FRt1/assignment_loss‚3.1421926282304993 0loss/FRt2/assignment_loss‚1.6990369341682656 .loss/FRt2/detection_loss‚1.360482978704465 &loss/total_loss‚13.044672716658118 _step‚93 2loss/SRqqt2/assignment_loss‚0.7851369997024521 0loss/FBt1/assignment_loss‚0.0968935976664041 1loss/SRqqt1/detection_loss‚1.3878622136639254 % _timestamp‚1.7842737917568786e+09 epoch‚2 1loss/FBt2/assignment_loss‚0.22979103183072563]ì%áÛ‚(:’ …·çÒÅú¤ gpu.0.gpu‚92.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•^áþÜ‚(:ó …·çÒ‰¥˜¦ disk.sda.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 250.1171875 proc.cpu.threads‚17 network.sent‚ 1.672899e+06 disk.dm-1.out‚ 28.92578125 disk.drbd2165.in‚0 network.recv‚304607$ disk./.usageGB‚549.2687225341797 disk.rbd1.out‚2.5625 disk.nvme0n1.in‚45.1875 cpu‚25.65049513745784 disk.dm-0.in‚45.1875! disk.dm-0.out‚5354.3779296875$ disk.nvme0n1.out‚5354.3779296875& disk.nvme0c0n1.out‚5354.3779296875 disk.sda2.out‚0 disk.sda3.out‚ 28.92578125) disk./.usagePercent‚36.86188483642388 disk.dm-1.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚45.1875 disk.sda2.in‚0# memory_percent‚98.4734058380127# proc.memory.rssMB‚ 2744.47265625 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0) proc.memory.percent‚16.75093173980713 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 28.92578125 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0U×GÁ“Ý‚(:ˆ ”·çÒÒ±æ¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@•ÛÊôÞ‚(:é ”·çÒÔˆ®§ network.recv‚309122 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.out‚2.5625# proc.memory.rssMB‚ 2744.47265625* disk./.usagePercent‚36.861898404823194 disk.rbd2.in‚0 network.sent‚ 1.72309e+06$ disk./.usageGB‚549.2689247131348 disk.sda1.in‚0 disk.dm-1.out‚29.625 proc.cpu.threads‚17 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 45.90234375 disk.rbd3.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 250.4140625 disk.nvme0n1.in‚ 45.90234375! disk.nvme0n1.out‚ 5364.1484375 disk.rbd0.in‚0 disk.sda.out‚29.625 disk.sda2.out‚0 disk.sda3.out‚29.625 cpu‚25.661137432179547 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0) proc.memory.percent‚16.75093173980713# disk.nvme0c0n1.out‚ 5364.1484375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 45.90234375 disk.dm-0.out‚ 5364.1484375 disk.rbd2.out‚0$ memory_percent‚98.47159385681152yDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252749432832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9403594 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268332882432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180916582817741 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚95 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11044279 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181837300613632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268258984960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180584192802816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181837300613632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9403592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4667451 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2104921 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2562533 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40270313310208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180584209842176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180916225228495 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179665399451648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9403594 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚73897472 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20447871 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41538553682944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2562791 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1250825776847 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181836942235648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11044279 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4667712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 # _timestamp‚1.78427383151086e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1253090771456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252749432832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181836942235648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚1023.636673409 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438ÞZbº¿æÐO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181836942235648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚1023.636673409 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252749432832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9403594 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268332882432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180916582817741 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚95 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11044279 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181837300613632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268258984960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180584192802816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181837300613632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9403592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4667451 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2104921 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2562533 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40270313310208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180584209842176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180916225228495 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179665399451648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OCDeviceStatsé úMonitor.on_train_batch_start/active.large_pool.allocated‚9403594 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚73897472 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20447871 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41538553682944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2562791 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1250825776847 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181836942235648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11044279 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4667712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 # _timestamp‚1.78427383151086e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1253090771456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252749432832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚96 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179811910123520 % _timestamp‚1.7842738321220348e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254103405568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11053331 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41573008432640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9411224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2564783 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181985195892224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1024.247814926 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9411226 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179811927162880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253762059776 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4671195 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269284976128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181064104543447 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11063321 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180731433832448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2564532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269365157888 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9411224 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181063746954089 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303742068736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303723456512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181985554277376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180731450871808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253762059776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚1199 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2106663 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180731433832448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20464555 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9411226 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1252177380567 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11063321 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41573107226624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3›-JÂâ&éÜM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181063746954089 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303742068736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303723456512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181985554277376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180731450871808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253762059776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚1199 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2106663 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180731433832448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20464555 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9411226 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1252177380567 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11063321 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41573107226624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚96 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179811910123520 % _timestamp‚1.7842738321220348e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254103405568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11053331 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41573008432640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9411224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2564783 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181985195892224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1024.247814926 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9411226 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179811927162880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253762059776 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4671195 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269284976128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181064104543447 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11063321 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180731433832448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2564532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269365157888 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9411224 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0`{Ô Ï2ê- _wandbruntime‚1024  _runtime‚1024qKpëÿ /loss/FBt2/detection_loss‚1.6709858937188984 _step‚97 0loss/FRt1/assignment_loss‚3.0855466375431946 1loss/FBt1/assignment_loss‚0.09625465025046105 /loss/FRt1/detection_loss‚0.5799507515686253 1loss/SRqqt1/detection_loss‚1.4024227366452795 /loss/FBt1/detection_loss‚0.8061472084873867 epoch‚2 % _timestamp‚1.7842738321254933e+09 _runtime‚1024.251137679 /loss/FRt2/assignment_loss‚1.557908026907677 1loss/SRqqt1/assignment_loss‚1.032755798814743 2loss/SRqqt2/assignment_loss‚0.7282805032030544 1loss/FBt2/assignment_loss‚0.20857669671240728 /loss/FRt2/detection_loss‚1.3866065311669267 1loss/SRqqt2/detection_loss‚0.8081767426257332 &loss/total_loss‚13.363612177644388 trainer/global_step‚1199ûÿŸ ìƒ % _timestamp‚1.7842738321254933e+09 _runtime‚1024.251137679 /loss/FRt2/assignment_loss‚1.557908026907677 1loss/SRqqt1/assignment_loss‚1.032755798814743 2loss/SRqqt2/assignment_loss‚0.7282805032030544 1loss/FBt2/assignment_loss‚0.20857669671240728 /loss/FRt2/detection_loss‚1.3866065311669267 1loss/SRqqt2/detection_loss‚0.8081767426257332 &loss/total_loss‚13.363612177644388 trainer/global_step‚1199 /loss/FBt2/detection_loss‚1.6709858937188984 _step‚97 0loss/FRt1/assignment_loss‚3.0855466375431946 1loss/FBt1/assignment_loss‚0.09625465025046105 /loss/FRt1/detection_loss‚0.5799507515686253 1loss/SRqqt1/detection_loss‚1.4024227366452795 /loss/FBt1/detection_loss‚0.8061472084873867 epoch‚2aö.þ–‘í‚(:† ²·çÒœàê¤ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c•‚ëûî‚(:ð ²·çÒï¦û© disk.rbd0.in‚0 network.sent‚ 1.746294e+06 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚47.75 disk.sda2.in‚0 disk.dm-0.in‚47.75) proc.memory.percent‚16.80734157562256 network.recv‚316029 disk.sda1.out‚0 disk.sda.out‚ 30.03515625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.711829786484895 disk.rbd0.out‚0* disk./.usagePercent‚36.615176263849804 disk.sda3.in‚0 disk.rbd1.out‚2.703125# memory_percent‚98.5297679901123 disk.rbd2.out‚0 disk.nvme0n1.in‚47.75 disk.sda.in‚0! disk.dm-0.out‚5381.5673828125' proc.memory.availableMB‚ 240.8828125 proc.cpu.threads‚17# disk./.usageGB‚545.592586517334 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚5381.5673828125 disk.dm-1.out‚ 30.03515625 disk.rbd2.in‚0$ disk.nvme0n1.out‚5381.5673828125 disk.sda3.out‚ 30.03515625 disk.dm-1.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2753.71484375G×Ý2ï- _wandbruntime‚1042  _runtime‚1042ðºÐ¬ðÆ{ <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0031164498068392277 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1200602650642395 particle/accuracy_2_of_3‚1 trainer/global_step‚1199 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929511547088623 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6518707871437073 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6406704783439636 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6406704783439636 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6130067110061646 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5535916090011597 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212964177131653 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 _runtime‚1042.166440218 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_2‚0.7331212759017944 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11777689307928085 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9839288592338562 particle/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07963662594556808 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5026859641075134 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6544946432113647 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.33424440026283264 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5356574058532715 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9872227907180786 +particle/specificity‚0.8936882019042969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8363919258117676 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6844021677970886 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5590002536773682 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6799864172935486 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9971707463264465 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1200602650642395 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 +jet/accuracy_3_of_4‚0.05895020440220833 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.296875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5364583730697632 _step‚98 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932644963264465 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4854816198348999 *jet/accuracy_2_of_4‚0.5896281599998474 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5987424850463867 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6830757856369019 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32900139689445496 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9848240613937378 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4854167103767395 particle/accuracy_2_of_4‚1 & P¢ËO~ -urity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950447082519531 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12324415892362595 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08284614235162735 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9924427270889282 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1067386046051979 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971634745597839 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7010771632194519 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9696482419967651 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 /particle/accuracy_3_of_3‚0.9985215067863464 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.912835955619812 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9971707463264465 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5657080411911011 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 *loss/val_total_loss‚12.511934019482387 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_1_of_5‚1 % _timestamp‚1.7842738500398269e+09 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43389466404914856 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10916150361299515 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09385815262794495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.09651875495910645 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13714805245399475 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23022320866584778 jet/accuracy_1_of_5‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32998964190483093 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5326288938522339 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23994939029216766 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42079100012779236 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10927031934261322 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9944444298744202 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14129389822483063 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.558817982673645 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6753210425376892 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5344802141189575 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11546351760625839 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6550902128219604 +jet/accuracy_1_of_1‚0.25212177634239197 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18519242107868195 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09555491805076599 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15829364955425262 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5404239296913147 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5915201902389526 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.46428629755973816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6544843316078186 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9960886836051941 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0284799225628376 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12999999523162842 'particle/f_score‚0.6052791476249695 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8227317333221436 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11512576788663864 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_1‚1 6validation_average_jet_accuracy‚0.3009500503540039 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12666870653629303 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48566973209381104 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978646039962769 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019462302327156067 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1256767213344574 )jet/accuracy_1_of_3‚0.869462251663208 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5439839363098145 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5522750020027161 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.24719016253948212 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8475383520126343 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06459351629018784 +jet/accuracy_3_of_3‚0.05659361556172371 jet/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10887719690799713 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.361884742975235 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6588308215141296 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,jet/accuracy_4_of_4‚0.007130124606192112 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872687458992004 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6140828132629395 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4997503161430359 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_2_of_3‚0.38822445273399353 jet/accuracy_4_of_5‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.695993959903717 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10015104711055756 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5643429160118103 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15024861693382263 (particle/accuracy‚0.8312059044837952 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6149283647537231 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 4 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9945108294487 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.09601708501577377 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4090355336666107 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0738992691040039 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09309079498052597 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6981627345085144 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 epoch‚2 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8797425627708435 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6798679828643799 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9914016723632812 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9918980598449707 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16762226819992065 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017575599253177643 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08035623282194138 /particle/accuracy_4_of_4‚0.9714725017547607 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5467633605003357 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3888888955116272 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6005101799964905 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6981859803199768 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1071019172668457 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5902979373931885 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9783884882926941 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5657080411911011 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6769168376922607 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0056561981327831745 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08296384662389755 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009196646511554718 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.10843094438314438 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +particle/sensitivity‚0.6027167439460754 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22491225600242615 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9805382490158081 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11819024384021759 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5687994956970215 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5710933804512024 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5995890498161316 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6879441142082214 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_4‚0.926999568939209 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08296384662389755Ð. Ð=ñÊ{ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5344802141189575 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11546351760625839 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6550902128219604 +jet/accuracy_1_of_1‚0.25212177634239197 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18519242107868195 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09555491805076599 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15829364955425262 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5404239296913147 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5915201902389526 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.46428629755973816 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6544843316078186 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9960886836051941 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0284799225628376 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12999999523162842 'particle/f_score‚0.6052791476249695 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8227317333221436 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11512576788663864 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_1‚1 6validation_average_jet_accuracy‚0.3009500503540039 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12666870653629303 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48566973209381104 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978646039962769 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.019462302327156067 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1256767213344574 )jet/accuracy_1_of_3‚0.869462251663208 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5439839363098145 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5522750020027161 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.24719016253948212 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8475383520126343 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06459351629018784 +jet/accuracy_3_of_3‚0.05659361556172371 jet/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10887719690799713 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.361884742975235 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6588308215141296 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,jet/accuracy_4_of_4‚0.007130124606192112 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872687458992004 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6140828132629395 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4997503161430359 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_2_of_3‚0.38822445273399353 jet/accuracy_4_of_5‚0 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.695993959903717 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10015104711055756 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5643429160118103 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15024861693382263 (particle/accuracy‚0.8312059044837952 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6149283647537231 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 4 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9945108294487 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.09601708501577377 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4090355336666107 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0738992691040039 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09309079498052597 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6981627345085144 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 epoch‚2 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8797425627708435 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6798679828643799 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9914016723632812 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9918980598449707 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16762226819992065 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017575599253177643 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08035623282194138 /particle/accuracy_4_of_4‚0.9714725017547607 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5467633605003357 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3888888955116272 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6005101799964905 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6981859803199768 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1071019172668457 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5902979373931885 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9783884882926941 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5657080411911011 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6769168376922607 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0056561981327831745 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08296384662389755 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009196646511554718 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.10843094438314438 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +particle/sensitivity‚0.6027167439460754 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22491225600242615 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9805382490158081 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11819024384021759 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5687994956970215 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5710933804512024 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5995890498161316 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6879441142082214 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_4‚0.926999568939209 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08296384662389755 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0031164498068392277 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1200602650642395 particle/accuracy_2_of_3‚1 trainer/global_step‚1199 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929511547088623 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6518707871437073 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6406704783439636 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6406704783439636 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6130067110061646 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5535916090011597 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212964177131653 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 _runtime‚1042.166440218 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_2‚0.7331212759017944 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11777689307928085 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9839288592338562 particle/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07963662594556808 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5026859641075134 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6544946432113647 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.33424440026283264 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5356574058532715 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9872227907180786 +particle/specificity‚0.8936882019042969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8363919258117676 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6844021677970886 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5590002536773682 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6799864172935486 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9971707463264465 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1200602650642395 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 +jet/accuracy_3_of_4‚0.05895020440220833 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.296875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5364583730697632 _step‚98 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9932644963264465 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4854816198348999 *jet/accuracy_2_of_4‚0.5896281599998474 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5987424850463867 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6830757856369019 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32900139689445496 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9848240613937378 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4854167103767395 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950447082519531 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12324415892362595 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08284614235162735 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9924427270889282 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1067386046051979 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971634745597839 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7010771632194519 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9696482419967651 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 /particle/accuracy_3_of_3‚0.9985215067863464 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.912835955619812 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9971707463264465 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5657080411911011 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 *loss/val_total_loss‚12.511934019482387 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_1_of_5‚1 % _timestamp‚1.7842738500398269e+09 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43389466404914856 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10916150361299515 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09385815262794495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.09651875495910645 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13714805245399475 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23022320866584778 jet/accuracy_1_of_5‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32998964190483093 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5326288938522339 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23994939029216766 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42079100012779236 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10927031934261322 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9944444298744202 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14129389822483063 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.558817982673645 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6753210425376892bTÜI—“ò‚(:ˆ Á·çÒÚdz¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÛzýó‚(:ò Á·çÒò룦 disk.sda2.out‚0 disk.dm-0.in‚47.75 disk.sda.out‚ 31.64453125 disk.sda2.in‚0 cpu‚25.73923254045972 disk.rbd2.in‚0 disk.nvme0n1.in‚47.75 disk.drbd2165.in‚0$ memory_percent‚96.96958065032959 network.recv‚320201) disk./.usagePercent‚36.63110530863777$ disk./.usageGB‚545.8299407958984' disk.nvme0c0n1.out‚5634.62353515625 disk.sda3.out‚ 31.64453125 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0" disk.dm-0.out‚5634.62353515625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚47.75 disk.rbd1.out‚2.765625 disk.rbd3.out‚0( proc.memory.percent‚16.8637752532959 disk.rbd2.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 31.64453125( proc.memory.availableMB‚ 496.50390625" proc.memory.rssMB‚ 2762.9609375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 1.809043e+06% disk.nvme0n1.out‚5634.62353515625 disk.drbd2165.out‚0·mÏ\“ô‚(:ˆ зçÒò¥â£ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†‰Îõ‚(:û зçÒæÌÃ¥ network.recv‚324169 disk.dm-1.out‚ 32.78515625 disk.rbd1.out‚2.765625 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 497.1171875& disk.nvme0c0n1.out‚5677.9736328125 disk.sda2.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚5677.9736328125) proc.memory.percent‚16.86379909515381 disk.nvme0c0n1.in‚ 48.578125 disk.sda3.out‚ 32.78515625 disk.nvme0n1.in‚ 48.578125 disk.sda1.out‚0# proc.memory.rssMB‚ 2762.96484375 network.sent‚ 1.855602e+06 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 48.578125 disk.drbd2165.in‚0$ disk./.usageGB‚545.8284111022949 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0! disk.dm-0.out‚5677.9736328125 disk.sda.out‚ 32.78515625 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 cpu‚25.74796848782583) disk./.usagePercent‚36.63100264961654 disk.sda1.in‚0# memory_percent‚96.9658374786377 proc.cpu.threads‚17:Z!œ“ö‚(:ˆ ß·çÒ•§©¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kA† ð÷‚(:å ß·çÒ·±Û§ disk.nvme0n1.out‚ 5899.34375 disk.dm-1.in‚0' proc.memory.availableMB‚ 497.1015625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 49.29296875! disk.nvme0c0n1.out‚ 5899.34375 disk.rbd1.out‚2.84375 cpu‚25.756553294340556 network.sent‚ 1.858566e+06 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.sda3.out‚33.4375) proc.memory.percent‚16.86382293701172 network.recv‚327911 disk.sda.in‚0* disk./.usagePercent‚36.630902038655584$ disk./.usageGB‚545.8269119262695 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 5899.34375 disk.sda1.in‚0 disk.sda.out‚33.4375 disk.dm-1.out‚33.4375 disk.nvme0n1.in‚ 49.29296875 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚96.96593284606934 proc.cpu.threads‚17 disk.dm-0.in‚ 49.29296875 disk.drbd2165.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2762.96875 disk.rbd2.in‚0à°ª©2ø- _wandbruntime‚1084  _runtime‚1084š´ÆùÀ _runtime‚1084.119776329 _step‚99 & lr-AdamW/pg1‚0.0004704833264085933 & lr-AdamW/pg2‚0.0004704833264085933 trainer/global_step‚1249 % _timestamp‚1.7842738919917517e+09;©­ÊúÄ % _timestamp‚1.7842738919917517e+09 _runtime‚1084.119776329 _step‚99 & lr-AdamW/pg1‚0.0004704833264085933 & lr-AdamW/pg2‚0.0004704833264085933 trainer/global_step‚1249c”Ç3ã2û- _wandbruntime‚1084  _runtime‚1084B¡ñüÍO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/se“;ç ¾%gment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191196989076480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚189875810934784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189875827974144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚189875810934784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1337517365760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11569056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11579033 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9905860 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9905858 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43295180702208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4890769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9905858 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41957663336448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41957644724224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190218758248312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9905860 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚189875827974144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11569056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2194691 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1319502538874 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319161988984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4890508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚1249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191197347454976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1321178141696 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11579033 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1321519480832 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2696078 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191197347454976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190219115837562 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191196989076480 # _timestamp‚1.78427389199592e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2194688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚1249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191197347454976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1321178141696 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11579033 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1321519480832 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2696078 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191197347454976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190219115837562 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191196989076480 # _timestamp‚1.78427389199592e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2194688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191196989076480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚189875810934784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189875827974144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚189875810934784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1337517365760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11569056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11579033 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9905860 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9905858 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43295180702208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4890769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9905858 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41957663336448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976d^˜<2þ- _wandbruntime‚1084  _runtime‚1084lœ¡ß&ÿÙM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11588099 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1338543255552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11578108 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190023069003776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43329530265600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320173042706 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9913490 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1322190768640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913492 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2698103 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1320513592708 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2196433 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41991073482752 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11588099 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9913490 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4894258 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚190366637563268 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4894539 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1322532114944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189046123970560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9913492 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190023069003776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191345601118720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1338456782848 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191345601118720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1322532114944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43329635350528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190023051964416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 trainer/global_step‚1249 _step‚101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41991092094976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2697825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191345242733056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190366279973906 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2196436 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1322190768640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872ÆmM7 €ÝM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚189046106931200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190023051964416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 trainer/global_step‚1249 _step‚101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41991092094976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2697825 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191345242733056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190366279973906 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2196436 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1322190768640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11588099 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1338543255552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11578108 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19002306900377uEÐy¬6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43329530265600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320173042706 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9913490 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1322190768640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913492 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2698103 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1320513592708 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2196433 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41991073482752 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11588099 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9913490 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4894258 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚190366637563268 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4894539 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1322532114944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189046123970560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9913492 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190023069003776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191345601118720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1338456782848 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191345601118720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1322532114944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43329635350528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190023051964416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198609736059904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12023508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22320945 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1369778475892 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚1124.454479901 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2281938 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197594908328820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22310966 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1372214982656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1388744171008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1372214982656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196225129852928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1371873643520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12033485 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10287458 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10287460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5081097 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚104 % _timestamp‚1.7842739323287292e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43629082650624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388655593984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45017845433856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198610094438400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2798887 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198609736059904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43629101262848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197237879455744 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5080825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10287460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101L¤#Þ'ØO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2798887 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198609736059904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43629101262848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197237879455744 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5080825 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10287460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12023508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚1299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197237879455744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198609736059904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12023508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22320945 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1369778475892 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚1124.454479901 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2281938 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197594908328820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22310966 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1372214982656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1388744171008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1372214982656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196225129852928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1371873643520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12033485 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10287458 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10287460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5081097 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚104 % _timestamp‚1.7842739323287292e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43629082650624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388655593984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45017845433856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198610094438400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101hÖ}û\¶Ž‚(:ˆ Œ¸çÒÊ®Š¥ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gp­"ûÝu.0.powerWatts‚265.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MôT ‚(:€ Œ¸çÒ™ûѦ* disk./.usagePercent‚36.630850325133665 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 34.5859375 disk.dm-1.in‚0 disk.dm-0.in‚ 50.8359375# proc.memory.rssMB‚ 2762.97265625) proc.memory.percent‚16.86384677886963 cpu‚25.778776619712495 disk.nvme0n1.in‚ 50.8359375% disk.nvme0n1.out‚5923.11767578125 disk.sda.in‚0 disk.sda.out‚ 34.5859375 network.recv‚339024 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚5923.11767578125 network.sent‚ 1.931636e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 50.8359375 disk.dm-1.out‚ 34.5859375" disk.dm-0.out‚5923.11767578125' proc.memory.availableMB‚ 496.8203125 disk.drbd2165.in‚0 disk.rbd1.out‚2.90625$ disk./.usageGB‚545.8261413574219 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚96.96764945983887a—T2- _wandbruntime‚1125  _runtime‚1125örÍã&‘ÝM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22337643 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1389675437056 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1372886270464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12032560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10295092 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5084884 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10295090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45052186846208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197742430054414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43662511409152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371130079616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _runtime‚ 1125.07478132 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1372886270464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197385103446016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12042551 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198757989716480 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5084629 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1370789529614 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198758348102144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚197385120485376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45052291931136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _step‚105 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10295092 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842739329490004e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚1299 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1373227616768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474Œ8ç&’áM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10295092 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842739329490004e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚1299 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1373227616768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22337643 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1389675437056 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1372886270464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12032560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10295092 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5084884 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10295090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45052186846208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197742430054414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43662511409152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371130079616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _runtime‚ 1125.07478132 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1372886270464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197385103446016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12042551 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198757989716480 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5084629 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1370789529614 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198758348102144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚197385120485376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45052291931136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _step‚105 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888il2“- _wandbruntime‚1125  _runtime‚1125CˆêS”ÿ &loss/total_loss‚12.726502027941851 /loss/FBt1/detection_loss‚0.8812881268047906 /loss/FBt2/detection_loss‚1.4587097734212875 _runtime‚ 1125.07591019 /loss/FRt1/detection_loss‚0.5883406859729773 % _timestamp‚1.7842739329505959e+09 0loss/FRt2/assignment_loss‚1.5281412197530062 1loss/FBt1/assignment_loss‚0.06392609850424878 1loss/SRqqt1/detection_loss‚1.4018214292982791 epoch‚3 trainer/global_step‚1299 _step‚106 0loss/FRt1/assignment_loss‚2.8724355239589525 2loss/SRqqt1/assignment_loss‚0.8894972369391867 1loss/SRqqt2/assignment_loss‚0.766151833178663 1loss/FBt2/assignment_loss‚0.12355377963607074 .loss/FRt2/detection_loss‚1.419357712116524 1loss/SRqqt2/detection_loss‚0.7332786083578655B´b •ƒ 1loss/SRqqt2/assignment_loss‚0.766151833178663 1loss/FBt2/assignment_loss‚0.12355377963607074 .loss/FRt2/detection_loss‚1.419357712116524 1loss/SRqqt2/detection_loss‚0.7332786083578655 &loss/total_loss‚12.726502027941851 /loss/FBt1/detection_loss‚0.8812881268047906 /loss/FBt2/detection_loss‚1.4587097734212875 _runtime‚ 1125.07591019 /loss/FRt1/detection_loss‚0.5883406859729773 % _timestamp‚1.7842739329505959e+09 0loss/FRt2/assignment_loss‚1.5281412197530062 1loss/FBt1/assignment_loss‚0.06392609850424878 1loss/SRqqt1/detection_loss‚1.4018214292982791 epoch‚3 trainer/global_step‚1299 _step‚106 0loss/FRt1/assignment_loss‚2.8724355239589525 2loss/SRqqt1/assignment_loss‚0.8894972369391867j·(Bâ“–‚(:ˆ ›¸çÒÓõ§¢ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07;ìi —‚(:ÿ ›¸çÒ¯±Ý§ cpu‚25.784899631755046 disk.rbd0.out‚0$ memory_percent‚96.96288108825684* disk./.usagePercent‚36.630887702233665 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.out‚2.984375 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚343628$ disk.nvme0n1.out‚5930.3447265625 disk.dm-1.out‚ 34.8046875 disk.rbd3.out‚0 network.sent‚ 1.983978e+06 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 34.8046875' proc.memory.availableMB‚ 497.6015625$ disk./.usageGB‚545.8266983032227 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚5930.3447265625 disk.sda2.out‚0 disk.sda3.out‚ 34.8046875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 51.6640625 disk.dm-0.in‚ 51.6640625# proc.memory.rssMB‚ 2762.17578125 disk.nvme0n1.in‚ 51.6640625 disk.sda3.in‚0! disk.dm-0.out‚5930.3447265625* proc.memory.percent‚16.858983039855957u¥ ¾ž˜‚(:“ ª¸çÒïøè¤ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/$å3 ™‚(:€ ª¸çÒ슧 disk.rbd1.out‚ 2.98828125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚5939.3017578125 disk.rbd0.in‚0 disk.sda.out‚ 34.984375# proc.memory.rssMB‚ 2762.97265625 network.sent‚ 2.038378e+06) disk./.usagePercent‚36.63059124550902 disk.rbd2.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚545.8222808837891 disk.rbd1.in‚0$ disk.nvme0n1.out‚5939.3017578125 disk.sda.in‚0 cpu‚25.791781070905394 proc.cpu.threads‚17 network.recv‚348104 disk.rbd2.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 34.984375 disk.nvme0n1.in‚ 52.37890625! disk.nvme0c0n1.in‚ 52.37890625$ memory_percent‚96.96877002716064 disk.dm-0.in‚ 52.37890625 disk.sda1.in‚0& disk.nvme0c0n1.out‚5939.3017578125 disk.sda1.out‚0 disk.sda3.out‚ 34.984375( proc.memory.availableMB‚ 496.63671875) proc.memory.percent‚16.86384677886963wäDT2š- _wandbruntime‚1164  _runtime‚1164¬SÏÆ›À trainer/global_step‚1349 $ _timestamp‚1.784273972795752e+09 _runtime‚1164.921347295 _step‚107 & lr-AdamW/pg1‚0.0004699393687804495 & lr-AdamW/pg2‚0.0004699393687804495.§VŽÊœÄ _runtime‚1164.921347295 _step‚107 & lr-AdamW/pg1‚0.0004699393687804495 & lr-AdamW/pg2‚0.0004699393687804495 trainer/global_step‚1349 $ _timestamp‚1.784273972795752e+09kŒ%.N2- _wandbruntime‚1164  _runtime‚1164…|qˆ+žÔO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204599930937344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12477960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204971415998578 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚206022841421824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2902291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12477960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1422910484480 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204599913897984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206022483043328 _step‚108 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45300520577024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204971058409328 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842739727968109e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5271482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204599930937344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203550663446528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚1164.922593598 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204599913897984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1420735512690 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439969710592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10669058 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203550680485888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206022483043328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10669060 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚206022841421824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5271207 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2902019 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2369191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1439887425024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚1349 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272»“­:Þ'ŸØO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23147018 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1422569145344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12487937 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46740508899840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204599913897984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206022483043328 _step‚108 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45300520577024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204971058409328 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842739727968109e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5271482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204599930937344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203550663446528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚1164.922593598 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204599913897984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1420735512690 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439969710592 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10669058 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203550680485888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206022483043328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10669060 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚206022841421824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5271207 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2902019 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2369191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1439887425024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚1349 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204599930937344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12477960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204971415998578 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚206022841421824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2902291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12477960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1422910484480 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440992471040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676692 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _runtime‚1165.533671183 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46774853236736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45333967947776 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10676690 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203697174118400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46774960418816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10676692 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚1349 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2904357 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206170736699904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1423581772288 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5275029 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12497003 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5275293 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440903901184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206171095085568 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12487012 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206171095085568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204747154927616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1421406016522 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45333949335552 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23163702 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚1349 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2904357 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206170736699904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1423581772288 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5275029 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12497003 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5275293 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440903901184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206171095085568 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12487012 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206171095085568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204747154927616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1421406016522 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45333949335552 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440992471040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676692 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _runtime‚1165.533671183 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46774853236736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚4533396Ið+®7947776 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10676690 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203697174118400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46774960418816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10676692 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695mlÏ*2£- _wandbruntime‚1165  _runtime‚1165í’ó¡¤ÿ &loss/total_loss‚13.174294493269105 epoch‚3 0loss/FRt1/assignment_loss‚2.8594752678037745 2loss/SRqqt1/assignment_loss‚0.8420304400045598 % _timestamp‚1.7842739734108334e+09 0loss/FBt2/assignment_loss‚0.4295825755103111 1loss/SRqqt2/detection_loss‚0.7841953327444771 trainer/global_step‚1349 .loss/FRt2/assignment_loss‚1.56673411702487 2loss/SRqqt2/assignment_loss‚0.7112713265323026 /loss/FRt1/detection_loss‚0.6334192131271427 1loss/SRqqt1/detection_loss‚1.2755346545235855 /loss/FBt1/detection_loss‚0.5779454762788696 /loss/FBt2/detection_loss‚1.9212070393065612 _runtime‚1165.536252225 1loss/FBt1/assignment_loss‚0.09517167493374228 /loss/FRt2/detection_loss‚1.4777273754789078 _step‚110ø-Ø ¥ƒ .loss/FRt2/assignment_loss‚1.56673411702487 2loss/SRqqt2/assignment_loss‚0.7112713265323026 /loss/FRt1/detection_loss‚0.6334192131271427 1loss/SRqqt1/detection_loss‚1.2755346545235855 /loss/FBt1/detection_loss‚0.5779454762788696 /loss/FBt2/detection_loss‚1.9212070393065612 _runtime‚1165.536252225 1loss/FBt1/assignment_loss‚0.09517167493374228 /loss/FRt2/detection_loss‚1.4777273754789078 _step‚110 &loss/total_loss‚13.174294493269105 epoch‚3 0loss/FRt1/assignment_loss‚2.8594752678037745 2loss/SRqqt1/assignment_loss‚0.8420304400045598 % _timestamp‚1.7842739734108334e+09 0loss/FBt2/assignment_loss‚0.4295825755103111 1loss/SRqqt2/detection_loss‚0.7841953327444771 trainer/global_step‚1349náa Ÿ¦‚(:’ ¹¸çÒÞ­å¥ gpu.0.gpu‚80.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H;H¡§‚(:… ¹¸çÒø¯·¨ disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚36.631032602498045$ disk.nvme0n1.out‚5946.0087890625 disk.drbd2165.in‚0 disk.rbd1.out‚ 3.11328125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 53.20703125& disk.nvme0c0n1.out‚5946.0087890625 disk.dm-0.in‚ 53.20703125( proc.memory.availableMB‚ 497.45703125 network.sent‚ 2.041171e+06 disk.sda2.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚5946.0087890625* proc.memory.percent‚16.859006881713867 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚96.96376323699951 proc.cpu.threads‚17 network.recv‚351111 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 35.23828125 disk.dm-1.in‚0 disk.dm-1.out‚ 35.23828125" proc.memory.rssMB‚ 2762.1796875# disk./.usageGB‚545.828857421875 disk.sda.in‚0 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 53.20703125 disk.sda.out‚ 35.23828125 disk.rbd1.in‚0 cpu‚25.79881540284094kùtÍ“¨‚(:ˆ ȸçÒóúú¤ gpu.0.gpu‚81.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V¯“J©‚(:„ ȸçÒàüù¦) disk./.usagePercent‚36.63100034554873$ disk./.usageGB‚545.8283767700195 disk.sda2.in‚0 disk.rbd1.out‚ 3.11328125$ memory_percent‚98.47099781036377( proc.memory.availableMB‚ 250.51171875" proc.memory.rssMB‚ 2762.9765625 disk.drbd2165.out‚0 disk.sda2.out‚0 network.recv‚355293 disk.nvme0n1.in‚ 78.9921875 disk.sda1.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚5950.06005859375 disk.sda.out‚ 35.45703125 disk.sda3.out‚ 35.45703125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 2.105474e+06 disk.rbd2.in‚0 disk.rbd2.out‚0% disk.nvme0n1.out‚5950.06005859375) proc.memory.percent‚16.86387062072754 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 78.9921875 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 35.45703125 disk.rbd0.in‚0 disk.dm-0.in‚ 78.9921875" disk.dm-0.out‚5950.06005859375 cpu‚25.80539572414891ß±Bžª‚(:“ ׸çÒšü«¤ gpu.0.gpu‚96.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vXë‚(:‡ ׸çÒˆ„û¥) disk./.usagePercent‚36.63101212189531 disk.dm-0.in‚ 79.70703125 disk.rbd0.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 251.3984375 disk.dm-1.in‚0 disk.dm-1.out‚ 35.83984375 proc.cpu.threads‚17" disk.dm-0.out‚5956.58349609375' disk.nvme0c0n1.out‚5956.58349609375* proc.memory.percent‚16.859006881713867 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 network.recv‚358888 disk.rbd0.out‚0" proc.memory.rssMB‚ 2762.1796875 cpu‚25.81219820161712 disk.sda1.out‚0 disk.sda.out‚ 35.83984375 disk.sda3.out‚ 35.83984375 disk.rbd1.out‚ 3.20703125$ memory_percent‚98.46558570861816 network.sent‚ 2.108497e+06 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 79.70703125 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0$ disk./.usageGB‚545.8285522460938 disk.rbd2.out‚0 disk.nvme0n1.in‚ 79.70703125% disk.nvme0n1.out‚5956.58349609375 disk.drbd2165.out‚0ÐÒþ:2¬- _wandbruntime‚1205  _runtime‚1205±Cb*Ç­Á _step‚111 & lr-AdamW/pg1‚0.0004696448749466644 & lr-AdamW/pg2‚0.0004696448749466644 trainer/global_step‚1399 % _timestamp‚1.7842740131117394e+09 _runtime‚1205.238572167pþAˮŠ& lr-AdamW/pg2‚0.0004696448749466644 trainer/global_step‚1399 % _timestamp‚1.7842740131117394e+09 _runtime‚1205.238572167 _step‚111 & lr-AdamW/pg1‚0.0004696448749466644o¤¥gp2¯- _wandbruntime‚1205  _runtime‚1205ŸC€ÃÚ'°ÔO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11050660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11050658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1491073055744 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1473605986304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12932412 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5461962 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48463140845568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212347566079086 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213435230026752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚112 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213435230026752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211961965379584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11050660 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11050658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211961982418944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5461694 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491163729920 _runtime‚1205.239562756 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211961965379584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3005256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46971977115648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12932412 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1473264647168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3005521 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1473605986304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12932412 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5461962 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48463140845568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212347566079086 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213435230026752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚112 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213435230026752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211961965379584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11050660 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11050658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211961982418944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5461694 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491163729920 _runtime‚1205.239562756 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211961965379584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3005256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46971977115648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12932412 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1473264647168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3005521 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3007302 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1472363053432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212109206409216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2458183 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47005405874176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1472022503430 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213583842068992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212109223448576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12951455 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11058290 VHDeviceStatsMonitor.on_train_batch_end/Ñšª/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1492181992960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213583842068992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474277274112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12941464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12951455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211022707712000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48497587867136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚113 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5465485 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7842740137277236e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12941464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212109206409216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474618620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474618620416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211022724751360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212109223448576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474277274112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11058292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11058290 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3007569 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212494730215430 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212495087804792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11058292€{˜0è&´âM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213583483683328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3007302 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1472363053432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚212109206409216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2458183 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47005405874176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1472022503430 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213583842068992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212109223448576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12951455 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11058290 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1492181992960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213583842068992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474277274112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12941464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12951455 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211022707712000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48497587867136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚113 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5465485 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7842740137277236e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12941464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212109206409216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474618620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474618620416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211022724751360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212109223448576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474277274112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11058292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11058290 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3007569 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212494730215430 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212495087804792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11058292 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1524301488128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218201747673088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13396841 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48643396429824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2543691 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219723358570344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3108316 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1524301488128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11432258 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11432258 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219324033900544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11432260 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2543688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1542285842944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1521968486506 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24819122 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1523960148992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220847977010176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219324016861184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1542380711424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5652280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220847977010176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220848335388672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13386864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50185795753472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚116 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5652004 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219723716159594 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220848335388672 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13386864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219324016861184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1521627936616 % _timestamp‚1.7842740535770445e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218201730633728 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1524301488128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218201747673088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13396841 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48643396429824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2543691 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219723358570344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3108316 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1524301488128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11432258 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11432258 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219324033900544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11432260 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2543688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1542285842944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1521968486506 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24819122 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1523960148992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220847977010176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219324016861184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1542380711424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5652280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220847977010176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220848335388672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13386864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50185795753472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚116 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5652004 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219723716159594 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220848335388672 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13386864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219324016861184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1521627936616 % _timestamp‚1.7842740535770445e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218201730633728 I2Ä- _wandbruntime‚1246  _runtime‚1246Ä#ä&ÅÞM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1524972775936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3110334 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219871237885300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219870880295938 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1543402140672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218348258344960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚117 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3110599 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48676825188352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1246.314974511 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220996589052416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220996230666752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50220245941248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219471257890816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219471274930176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1525314122240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220996589052416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5655767 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24835806 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13395916 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2545433 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218348241305600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1522638990338 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11439892 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50220142953472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169å„mè&ÆâM NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5656035 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚117 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3110599 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48676825188352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1246.314974511 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220996589052416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220996230666752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50220245941248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219471257890816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219471274930176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1525314122240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220996589052416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5655767 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24835806 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13395916 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2545433 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218348241305600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1522638990338 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24845799 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11439892 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50220142953472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1524972775936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3110334 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219871237885300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219870880295938 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1543402140672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218348258344960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226686068342784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3211528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228261082372096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚228261082372096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3211793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13841316 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51908378915840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1574996989952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228260723993600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226686085382144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228260723993600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225527281266688 _step‚120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1572584973414 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1574655650816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1593525947392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225527264227328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 trainer/global_step‚1499 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226686068342784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3211528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228261082372096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚228261082372096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3211793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13841316 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51908378915840 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1574996989952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228260723993600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226686085382144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228260723993600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225527281266688 _step‚120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1572584973414 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1574655650816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1593525947392 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225527264227328 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 trainer/global_step‚1499 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228408977650176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51942824903168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1573255477246 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1575668277760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11821492 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13850368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2632686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227247030376446 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51942707235328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225673774899200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25681851 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11821490 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228409336035840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228409336035840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1499 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚225673791938560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11821490 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13860359 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226833309372416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5846554 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3213868 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1594444120576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5846282 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1594543176192 _runtime‚1286.807605406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13860359 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _step‚121 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1575668277760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7842740946817741e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25671858 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1576009624064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227247387965808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1573596027248"mßÖâM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1575668277760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11821492 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13850368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2632686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_vbíÅ %streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227247030376446 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51942707235328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225673774899200 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25681851 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11821490 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228409336035840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228409336035840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1499 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚225673791938560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11821490 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13860359 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226833309372416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5846554 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3213868 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1594444120576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5846282 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1594543176192 _runtime‚1286.807605406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13860359 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _step‚121 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1575668277760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7842740946817741e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25671858 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1576009624064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227247387965808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1573596027248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2632683 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226833326411776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226833326411776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226833309372416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25671858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228408977650176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51942824903168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1573255477246 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976yENÖw2×- _wandbruntime‚1286  _runtime‚1286CÙ‡@Ø‚ 0loss/FRt1/assignment_loss‚2.8805603786971834 2loss/SRqqt1/assignment_loss‚0.8775268088918928 1loss/FBt2/assignment_loss‚0.13402980417984053 1loss/SRqqt2/detection_loss‚0.8052824596662308 /loss/FBt1/detection_loss‚0.4361147676540288 &loss/total_loss‚12.401317873954547 trainer/global_step‚1499 % _timestamp‚1.7842740946842723e+09 0loss/FRt2/assignment_loss‚1.6799682781154068 2loss/SRqqt2/assignment_loss‚0.6930351988513674 /loss/FRt1/detection_loss‚0.5945068314444102 1loss/SRqqt1/detection_loss‚1.4126039123272194 /loss/FBt2/detection_loss‚1.3519546031951903 epoch‚3 _runtime‚1286.809709769 _step‚122 /loss/FRt2/detection_loss‚1.3991830443428823 1loss/FBt1/assignment_loss‚0.13655178658889358»røp Ù† 1loss/FBt2/assignment_loss‚0.13402980417984053 1loss/SRqqt2/detection_loss‚0.8052824596662308 /loss/FBt1/detection_loss‚0.4361147676540288 &loss/total_loss‚12.401317873954547 trainer/global_step‚1499 % _timestamp‚1.7842740946842723e+09 0loss/FRt2/assignment_loss‚1.6799682781154068 2loss/SRqqt2/assignment_loss‚0.6930351988513674 /loss/FRt1/detection_loss‚0.5945068314444102 1loss/SRqqt1/detection_loss‚1.4126039123272194 /loss/FBt2/detection_loss‚1.3519546031951903 epoch‚3 _runtime‚1286.809709769 _step‚122 /loss/FRt2/detection_loss‚1.3991830443428823 1loss/FBt1/assignment_loss‚0.13655178658889358 0loss/FRt1/assignment_loss‚2.8805603786971834 2loss/SRqqt1/assignment_loss‚0.8775268088918928zw÷“Ú‚(:ˆ ±¹çÒ·ïÑ¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o¹r×Û‚(:ú ±¹çÒ·õä§ network.sent‚ 2.254693e+06 network.recv‚381458$ disk./.usageGB‚545.8306465148926 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 6003.7109375 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 37.47265625 disk.rbd2.out‚0# memory_percent‚98.4659194946289 disk.sda2.in‚0 disk.nvme0n1.in‚ 106.1328125! disk.nvme0n1.out‚ 6003.7109375 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 106.1328125* disk./.usagePercent‚36.631152670031604# proc.memory.rssMB‚ 2762.18359375 cpu‚25.84761675204782 proc.cpu.threads‚17 disk.dm-1.out‚ 37.47265625 disk.sda1.in‚0 disk.drbd2165.out‚0% proc.memory.availableMB‚ 251.34375* proc.memory.percent‚16.859030723571777 disk.sda3.out‚ 37.47265625 disk.dm-0.in‚ 106.1328125 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 6003.7109375 disk.rbd1.out‚ 3.41015625¬kd'›Ü‚(: À¹çÒ¹ôñ¤ gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.960000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(цÝ‚(:‰ À¹çÒ•¨¨§ disk.sda3.out‚ 37.88671875 disk.dm-1.out‚ 37.88671875) proc.memory.percent‚16.86387062072754 cpu‚25.85377124437251 network.recv‚386056 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 37.88671875 disk.sda2.in‚0$ memory_percent‚98.47054481506348" proc.memory.rssMB‚ 2762.9765625 network.sent‚ 2.319824e+06) disk./.usagePercent‚36.63116470238571$ disk./.usageGB‚545.8308258056641" disk.nvme0c0n1.in‚ 106.84765625 disk.dm-1.in‚0' proc.memory.availableMB‚ 250.5859375 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 106.84765625% disk.nvme0n1.out‚6012.43603515625 proc.cpu.threads‚17 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 106.84765625 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda3.in‚0" disk.dm-0.out‚6012.43603515625 disk.rbd1.out‚ 3.41015625 disk.sda1.in‚0' disk.nvme0c0n1.out‚6012.43603515625I$²ižÞ‚(:“ ϹçÒ‘€ò¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ýFí ß‚(:€ ϹçÒº‚²§ disk.sda1.out‚0 network.sent‚ 2.322746e+06* disk./.usagePercent‚36.631166494438446 disk.sda.out‚ 38.37109375 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚107.3125 disk.nvme0n1.in‚107.3125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚6017.52197265625 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.48828125" disk.dm-0.out‚6017.52197265625 disk.rbd2.in‚0% disk.nvme0n1.out‚6017.52197265625( proc.memory.availableMB‚ 250.58203125 disk.dm-1.in‚0 disk.dm-1.out‚ 38.37109375 cpu‚25.858852934254692 proc.cpu.threads‚17 network.recv‚389513$ disk./.usageGB‚545.8308525085449 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚107.3125 disk.rbd3.in‚0 disk.rbd2.out‚0 disk.rbd0.out‚0$ memory_percent‚98.47056865692139) proc.memory.percent‚16.86387062072754 disk.sda3.out‚ 38.37109375 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2762.9765625>22à- _wandbruntime‚1326  _runtime‚1326ÁÿiÈá $ _timestamp‚1.784274134409011e+09 _runtime‚1326.534560212 _step‚123 ' lr-AdamW/pg1‚0.00046867156322761156 ' lr-AdamW/pg2‚0.00046867156322761156 trainer/global_step‚1549Ð ö3ÌâÆ _runtime‚1326.534560212 _step‚123 ' lr-AdamW/pg1‚0.00046867156322761156 ' lr-AdamW/pg2‚0.00046867156322761156 trainer/global_step‚1549 $ _timestamp‚1.784274134409011e+09{zt§x2ã- _wandbruntime‚1326  _runtime‚1326©ÆÚ'äÔO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235673829355520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3315648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234476016320610 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14295768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1625692491776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1622860910432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12195460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12195458 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14305745 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623201460322 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235673829355520 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1625351152640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234475658731360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _runtime‚1326.535865065 _step‚124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53630961036800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51986290894848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51986272282624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12195458 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1644670141952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234048136863744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1625351152640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12195460 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2718188 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53630858041856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14305745 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3315383 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235673470977024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6033571 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1644585759232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚234048119824384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1625692491776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚1549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235673470977024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234048119824384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232852814860288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976!^îåØO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232852797820928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1625351152640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚234475658731360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _runtime‚1326.535865065 _step‚124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53630961036800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51986290894848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51986272282624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12195458 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1644670141952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234048136863744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1625351152640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12195460 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2718188 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53630858041856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14305745 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3315383 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235673470977024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6033571 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1644585759232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚234048119824384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1625692491776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚1549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235673470977024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234048119824384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232852814860288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235673829355520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3315648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234476016320610 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14295768 [IDeviïR¸_ceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1625692491776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1622860910432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12195460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12195458 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14305745 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623201460322 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235673829355520 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 1327.14989631 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234195377893376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14314811 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14304820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234195360854016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3317408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1645694303232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26517903 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12203090 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234195360854016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12203092 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842741350240798e+09 _step‚125 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645593150464 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235822083019264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1626363779584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235822083019264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1626363779584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1624212514156 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1623871964154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6037626 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234623180456954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6037341 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12203090 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234623538046316 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52019719653376 trainer/global_step‚1549 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2719936DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234195360854016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3317408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1645694303232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26517903 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12203090 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234195360854016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12203092 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842741350240798e+09 _step‚125 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645593150464 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235822083019264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1626363779584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235822083019264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1626363779584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1624212514156 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1623871964154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6037626 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234623180456954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6037341 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12203090 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234623538046316 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52019719653376 trainer/global_step‚1549 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2719936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 1327.14989631 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234195377893376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14314811 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14304820 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243086217960448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14760197 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6224673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚243086217960448 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3418969 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53657710209024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1695808623616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_¯a©Ðalloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241410188345344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240178331414528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241410188345344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3419232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55353434449920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53657728821248 _step‚128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12577060 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1695724240896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1673477397340 trainer/global_step‚1599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12577060 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1676387993600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚1367.082816865 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243086576338944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1676046654464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6224407 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2805438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1676046654464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6224407 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2805438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243086217960448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14760197 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6224673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚243086217960448 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3418969 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53657710209024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1695808623616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241410188345344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240178331414528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241410188345344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3419232 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55353434449920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53657728821248 _step‚128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12577060 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1695724240896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1673477397340 trainer/global_step‚1599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12577060 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1676387993600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚1367.082816865 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243086576338944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2807183 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚129 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚241557412335616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1696753062400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1677059281408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚243234830002688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3421065 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53691157579776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12584690 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55387892029952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1599 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14769263 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241557429374976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27343962 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241557429374976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12584692 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14759272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53691138967552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1674488451062 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1677400627712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14759272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1677059281408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 % _timestamp‚1.7842741755720181e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1674829001064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1677400627712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14769263 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6228248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55387995017728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243234471617024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3421319 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1696837437952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241999330537462 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240324842086400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2807183 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚129 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚241557412335616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1696753062400 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1677059281408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚243234830002688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3421065 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53691157579776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12584690 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55387892029952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1599 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14769263 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241557429374976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2ã×âÍÑ7343962 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241557429374976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12584692 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14759272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53691138967552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1674488451062 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1677400627712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14759272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1677059281408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 % _timestamp‚1.7842741755720181e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1674829001064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1677400627712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14769263 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6228248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55387995017728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243234471617024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3421319 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1696837437952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241999330537462 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169v %2û- _wandbruntime‚1367  _runtime‚1367c±]Ïü $ _timestamp‚1.784274175574044e+09 _runtime‚1367.699426997 /loss/FRt1/assignment_loss‚2.699481679319168 0loss/FBt1/detection_loss‚0.46129833757746397 &loss/total_loss‚11.150010338445417 2loss/SRqqt2/assignment_loss‚0.6584767875539393 /loss/FRt1/detection_loss‚0.6018724590339151 1loss/SRqqt2/detection_loss‚0.6969463283474774 epoch‚3 _step‚130 0loss/FRt2/assignment_loss‚1.3517956993236753 2loss/SRqqt1/assignment_loss‚0.6806989087220932 1loss/FBt1/assignment_loss‚0.09609641067901215 1loss/FBt2/assignment_loss‚0.12732999089386965 /loss/FRt2/detection_loss‚1.3776851893888917 /loss/FBt2/detection_loss‚1.1730765365064144 trainer/global_step‚1599 1loss/SRqqt1/detection_loss‚1.2252520110994969Ñ Œ ý† 2loss/SRqqt2/assignment_loss‚0.6584767875539393 /loss/FRt1/detection_loss‚0.6018724590339151 1loss/SRqqt2/detection_loss‚0.6969463283474774 epoch‚3 _step‚130 0loss/FRt2/assignment_loss‚1.3517956993236753 2loss/SRqqt1/assignment_loss‚0.6806989087220932 1loss/FBt1/assignment_loss‚0.09609641067901215 1loss/FBt2/assignment_loss‚0.12732999089386965 /loss/FRt2/detection_loss‚1.3776851893888917 /loss/FBt2/detection_loss‚1.1730765365064144 trainer/global_step‚1599 1loss/SRqqt1/detection_loss‚1.2252520110994969 $ _timestamp‚1.784274175574044e+09 _runtime‚1367.699426997 /loss/FRt1/assignment_loss‚2.699481679319168 0loss/FBt1/detection_loss‚0.46129833757746397 &loss/total_loss‚11.150010338445417‚À¡¤Õþ‚(:’ ‹ºçÒ¦òÁ¤ gpu.0.gpu‚58.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎkµEÿ‚(:ƒ ‹ºçÒŽ÷ì¥" disk.nvme0c0n1.in‚ 109.93359375 disk.sda2.out‚0 disk.dm-1.out‚ 39.3671875 disk.rbd3.out‚0 disk.drbd2165.in‚0( disk./.usagePercent‚36.6311690545138 disk.dm-0.in‚ 109.93359375 disk.rbd2.in‚0 disk.sda3.in‚0) proc.memory.percent‚16.89612865447998 network.recv‚404209 proc.cpu.threads‚17 disk.rbd0.out‚0$ memory_percent‚98.50373268127441' proc.memory.availableMB‚ 245.1484375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚6049.3056640625 disk.nvme0n1.in‚ 109.93359375$ disk.nvme0n1.out‚6049.3056640625 disk.sda1.in‚0# proc.memory.rssMB‚ 2768.26171875$ disk./.usageGB‚545.8308906555176 disk.rbd1.in‚0 disk.rbd2.out‚0& disk.nvme0c0n1.out‚6049.3056640625 disk.sda2.in‚0 disk.sda3.out‚ 39.3671875 cpu‚25.90764258590864 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda.out‚ 39.3671875 network.sent‚ 2.413946e+06 disk.rbd1.out‚ 3.69140625 disk.sda.in‚0¯Ö”ç2€ - _wandbruntime‚1385  _runtime‚1385L_Η²= ¬{ 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7008351683616638 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5445172190666199 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21518893539905548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7382897138595581 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1649693101644516 /particle/accuracy_3_of_4‚0.9946429133415222 trainer/global_step‚1599 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9846222400665283 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5800727009773254 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4012501537799835 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011180886998772621 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21525253355503082 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6861483454704285 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.034139879047870636 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026390448212623596 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_3_of_3‚0.9994933605194092 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9879439473152161 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852908253669739 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1566222459077835 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5066667199134827 jet/accuracy_1_of_5‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7676871418952942 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.05576518550515175 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.28900715708732605 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 _runtime‚1385.648013859 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6382114887237549 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.6866733431816101 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083432674408 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3453480899333954 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 *particle/specificity‚0.904110848903656 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9914641380310059 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6308215260505676 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923197627067566 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7163285613059998 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4560185670852661 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3564814627170563 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959602355957031 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6864957213401794 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.705564022064209 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6125472187995911 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1297832727432251 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7786093354225159 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01812860183417797 *loss/val_total_loss‚11.233156177492797 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1575053632259369 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2335028350353241 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08856935799121857 jet/accuracy_3_of_5‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5748812556266785 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7651875019073486 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46791940927505493 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.29316315054893494 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5499075055122375 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6734316945075989 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6097750067710876 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959602355957031 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.301777184009552 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1360500156879425 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7011094093322754 particle/accuracy_1_of_1‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6414278745651245 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.632113516330719 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8927931785583496 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5712278485298157 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23785535991191864 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5887120962142944 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 *jet/accuracy_1_of_4‚0.9624844789505005 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 &particle/f_score‚0.595849335193634 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08856935799121857 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17032143473625183 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3828125 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6490685343742371 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17089970409870148 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 particle/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952305555343628 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944060444831848 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9929102659225464 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8330388069152832 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.819468080997467 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5740227699279785 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46402040123939514 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3432731032371521 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3171790540218353 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_4_of_4‚0.9217609167098999 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16904178261756897 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020388374105095863 6validation_average_jet_accuracy‚0.3342357575893402 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.130033940076828 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20047785341739655 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.7571970224380493 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 _step‚131 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.5731768608093262 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5230178833007812 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5782424211502075 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.177481010556221 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1839444935321808 $ _timestamp‚1.784274193521703e+09 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13432034850120544 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0950617641210556 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6375250220298767 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.033896394073963165 *jet/accuracy_2_of_3‚0.4449427127838135 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17089970409870148 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09409834444522858 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9928113222122192 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.345215767621994 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8309088945388794 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.993941605091095 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952780604362488 ,jet/accuracy_4_of_4‚0.009931244887411594 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5896196961402893 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5275229811668396 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1649477779865265 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7651875019073486 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_4_of_5‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18320830166339874 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16446688771247864 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5896196961402893 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1394653022289276 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7815003991127014 +jet/accuracy_2_of_2‚0.11136806756258011 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6377004981040955 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9137429594993591 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847441911697388 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6066861152648926 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9705815315246582 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12166666984558105 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6207849383354187 *jet/accuracy_1_of_3‚0.8987535834312439 epoch‚3 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5967232584953308 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17052599787712097 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6114877462387085 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5816370248794556 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12375408411026001 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1324937641620636 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5594494342803955 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991701602935791 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966583251953125 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9926689863204956 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7334027290344238 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.180667445063591 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7195726037025452 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2721443474292755 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6595972180366516 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09404613077640533 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_1_of_1‚0.29213422536849976 particle/accuracy_2_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7388381958007812 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15025828778743744 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09533684700727463 +jet/accuracy_3_of_4‚0.05224595218896866 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18296892940998077 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8343571424484253 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7725623846054077 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645….ˆ"‚ ±{ 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6382114887237549 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.6866733431816101 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083432674408 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3453480899333954 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 *particle/specificity‚0.904110848903656 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9914641380310059 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6308215260505676 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923197627067566 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7163285613059998 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4560185670852661 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3564814627170563 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959602355957031 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6864957213401794 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.705564022064209 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6125472187995911 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1297832727432251 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7786093354225159 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01812860183417797 *loss/val_total_loss‚11.233156177492797 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1575053632259369 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2335028350353241 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08856935799121857 jet/accuracy_3_of_5‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5748812556266785 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7651875019073486 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46791940927505493 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.29316315054893494 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5499075055122375 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6734316945075989 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6097750067710876 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959602355957031 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.301777184009552 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1360500156879425 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7011094093322754 particle/accuracy_1_of_1‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6414278745651245 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.632113516330719 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8927931785583496 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5712278485298157 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23785535991191864 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5887120962142944 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 *jet/accuracy_1_of_4‚0.9624844789505005 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 &particle/f_score‚0.595849335193634 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08856935799121857 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17032143473625183 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3828125 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6490685343742371 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17089970409870148 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 particle/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952305555343628 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944060444831848 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9929102659225464 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8330388069152832 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.819468080997467 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5740227699279785 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46402040123939514 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3432731032371521 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3171790540218353 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_4_of_4‚0.9217609167098999 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16904178261756897 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020388374105095863 6validation_average_jet_accuracy‚0.3342357575893402 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.130033940076828 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20047785341739655 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.7571970224380493 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 _step‚131 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.5731768608093262 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5230178833007812 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5782424211502075 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.177481010556221 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1839444935321808 $ _timestamp‚1.784274193521703e+09 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13432034850120544 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0950617641210556 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6375250220298767 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.033896394073963165 *jet/accuracy_2_of_3‚0.4449427127838135 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17089970409870148 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09409834444522858 7 Purity/0FRt1SRqqt1FBt/FBt_pur¶¼ÔJ›ity‚0.9928113222122192 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.345215767621994 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8309088945388794 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.993941605091095 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952780604362488 ,jet/accuracy_4_of_4‚0.009931244887411594 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5896196961402893 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5275229811668396 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1649477779865265 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7651875019073486 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_4_of_5‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18320830166339874 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16446688771247864 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5896196961402893 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1394653022289276 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7815003991127014 +jet/accuracy_2_of_2‚0.11136806756258011 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6377004981040955 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9137429594993591 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847441911697388 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6066861152648926 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9705815315246582 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12166666984558105 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6207849383354187 *jet/accuracy_1_of_3‚0.8987535834312439 epoch‚3 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5967232584953308 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17052599787712097 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6114877462387085 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5816370248794556 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12375408411026001 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1324937641620636 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5594494342803955 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991701602935791 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966583251953125 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9926689863204956 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7334027290344238 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.180667445063591 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7195726037025452 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2721443474292755 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6595972180366516 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09404613077640533 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_1_of_1‚0.29213422536849976 particle/accuracy_2_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7388381958007812 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15025828778743744 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09533684700727463 +jet/accuracy_3_of_4‚0.05224595218896866 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18296892940998077 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8343571424484253 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7725623846054077 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7008351683616638 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5445172190666199 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21518893539905548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7382897138595581 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1649693101644516 /particle/accuracy_3_of_4‚0.9946429133415222 trainer/global_step‚1599 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9846222400665283 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5800727009773254 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4012501537799835 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011180886998772621 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21525253355503082 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6861483454704285 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.034139879047870636 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026390448212623596 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_3_of_3‚0.9994933605194092 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9879439473152161 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852908253669739 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1566222459077835 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5066667199134827 jet/accuracy_1_of_5‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7676871418952942 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.05576518550515175 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.28900715708732605 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 _runtime‚1385.648013859ƒ‹#Gžƒ ‚(:“ šºçұΤ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.875' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.775000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 !þþ„ ‚(:ƒ šºçÒˆ¼á¦ disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0$ memory_percent‚95.47662734985352$ disk./.usageGB‚545.8317337036133" disk.nvme0c0n1.in‚ 109.93359375 disk.dm-0.out‚6299.783203125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 2.46392e+06% disk.nvme0c0n1.out‚6299.783203125 disk.rbd1.out‚ 3.69140625 disk.dm-0.in‚ 109.93359375 disk.rbd2.out‚0 disk.nvme0n1.in‚ 109.93359375 disk.sda2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2776.18359375 disk.sda3.out‚ 39.53515625 disk.rbd1.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 741.109375 disk.sda.out‚ 39.53515625 disk.sda1.in‚0 disk.dm-1.out‚ 39.53515625 network.recv‚408465) disk./.usagePercent‚36.63122563217886 disk.sda2.out‚0 disk.rbd3.out‚0* proc.memory.percent‚16.944479942321777 cpu‚25.92228067676987 disk.sda.in‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚6299.783203125€ôã… ‚(:’ ©ºçÒ¿ºÐ£ gpu.0.gpu‚45.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P š † ‚(:€ ©ºçÒÏžò¥! disk.nvme0n1.out‚ 6554.0859375 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.75390625$ disk./.usageGB‚545.8307800292969 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 110.76171875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 110.76171875 disk.dm-0.out‚ 6554.0859375 cpu‚25.927064471481216 network.recv‚412395( disk./.usagePercent‚36.6311616302953 disk.sda1.in‚0 disk.sda3.out‚ 39.83203125 disk.rbd3.out‚0* proc.memory.percent‚16.944479942321777 disk.dm-1.out‚ 39.83203125 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 39.83203125 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 741.16015625 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.in‚0 network.sent‚ 2.523386e+06 disk.rbd2.out‚0 disk.nvme0n1.in‚ 110.76171875# disk.nvme0c0n1.out‚ 6554.0859375 disk.rbd1.in‚0$ memory_percent‚95.47631740570068# proc.memory.rssMB‚ 2776.18359375(Ÿj*ž‡ ‚(:“ ¸ºçÒñ·á¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚262.342' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.468399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 œ½ ˆ ‚(:þ ¸ºçÒ¥Ü㥠disk.sda3.out‚ 40.0703125 disk.sda2.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 2.528473e+06* disk./.usagePercent‚36.631083804004895 disk.rbd2.out‚0$ disk.nvme0c0n1.out‚ 6564.48046875 disk.sda.in‚0 disk.nvme0n1.in‚ 111.4765625 disk.sda1.out‚0 disk.dm-1.out‚ 40.0703125* proc.memory.percent‚16.944479942321777 disk.drbd2165.out‚0 disk.dm-0.in‚ 111.4765625 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0$ memory_percent‚95.47746181488037 cpu‚25.93121234884352 disk.dm-0.out‚ 6564.48046875 disk.rbd0.out‚0 disk.rbd1.out‚ 3.75390625 network.recv‚415430$ disk./.usageGB‚545.8296203613281" disk.nvme0n1.out‚ 6564.48046875 disk.rbd0.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2776.18359375 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 111.4765625 disk.rbd1.in‚0( proc.memory.availableMB‚ 740.97265625 disk.drbd2165.in‚0 disk.sda.out‚ 40.0703125FK¤¶2‰ - _wandbruntime‚1427  _runtime‚1427‹ûÌÆŠ À _step‚132 & lr-AdamW/pg1‚0.0004679480410128138 & lr-AdamW/pg2‚0.0004679480410128138 trainer/global_step‚1649 $ _timestamp‚1.784274235554933e+09 _runtime‚1427.680525158Ýù‰³Ë‹ Å & lr-AdamW/pg1‚0.0004679480410128138 & lr-AdamW/pg2‚0.0004679480410128138 trainer/global_step‚1649 $ _timestamp‚1.784274235554933e+09 _runtime‚1427.680525158 _step‚132„ÚÒi2Œ - _wandbruntime‚1427  _runtime‚1427êÈìÚ' ÔO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251154341831690 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252446264802304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744475364352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13079324 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚1427.681740272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6448560 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252446264802304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1649 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249412528222208 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252446623180800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250701806477312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744816703488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079326 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079324 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1742154159372 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15284976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15274999 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55345078847488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1765153782272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3553589 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250701806477312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2895211 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744475364352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13079326 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6448800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250701789437952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252446623180800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1741813609482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚133 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15284976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57110123343360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250701789437952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252446623180800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1741813609482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚133 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15284976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57110123343360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252446264802304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1649 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249412528222208 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252446623180800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250701806477312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744816703488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079326 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079324 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1742154159372 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15284976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15274999 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55345078847488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1765153782272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3553589 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088…ȶÏ2 - _wandbruntime‚1428  _runtime‚1428©"¸f! ÞM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252594518458880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/alLD‰›Ã%located_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3555452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15294042 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249559038894080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6452663 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086956 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745829337600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1745829337600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15284051 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2896956 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250849030467584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55378507606016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251301863557284 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2896953 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252594876844544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3555707 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252594876844544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1766176249856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚134 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1745487991296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7842742361682165e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1745487991296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251302221146646 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15284051 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1766083485696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250849030467584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13086956 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1428.294028478 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚250849047506944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6452405 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55378488993792 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3555452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15294042 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28371007 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249559038894080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6452663 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086956 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745829337600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1745829337600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15284051 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2896956 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250849030467584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55378507606016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251301863557284 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2896953 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252594876844544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3555707 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252594876844544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1766176249856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚134 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1745487991296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7842742361682165e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1745487991296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251302221146646 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15284051 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1766083485696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250849030467584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13086956 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1428.294028478 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚250849047506944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28371007 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6452405 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976†áàE)2’ - _wandbruntime‚1428  _runtime‚1428N(¨C“ þ 2loss/SRqqt1/assignment_loss‚0.8312514782293656 1loss/FBt2/assignment_loss‚0.20092970695824952 /loss/FRt1/detection_loss‚0.5551704697531173 /loss/FBt2/detection_loss‚1.4167072668119713 % _timestamp‚1.7842742361713567e+09 epoch‚4 0loss/FRt2/assignment_loss‚1.4022398480891358 2loss/SRqqt2/assignment_loss‚0.7394615533327032 1loss/FBt1/assignment_loss‚0.07917227507068263 /loss/FRt2/detection_loss‚1.3280791547193653 /loss/FBt1/detection_loss‚0.5631900553015208 1loss/SRqqt1/detection_loss‚1.1927869731136658 trainer/global_step‚1649 _step‚135 0loss/FRt1/assignment_loss‚2.5989836291655113 1loss/SRqqt2/detection_loss‚0.7155180533465713 %loss/total_loss‚11.62349046389186 _runtime‚ 1428.296694†]ò ” ƒ 2loss/SRqqt2/assignment_loss‚0.7394615533327032 1loss/FBt1/assignment_loss‚0.07917227507068263 /loss/FRt2/detection_loss‚1.3280791547193653 /loss/FBt1/detection_loss‚0.5631900553015208 1loss/SRqqt1/detection_loss‚1.1927869731136658 trainer/global_step‚1649 _step‚135 0loss/FRt1/assignment_loss‚2.5989836291655113 1loss/SRqqt2/detection_loss‚0.7155180533465713 %loss/total_loss‚11.62349046389186 _runtime‚ 1428.296694 2loss/SRqqt1/assignment_loss‚0.8312514782293656 1loss/FBt2/assignment_loss‚0.20092970695824952 /loss/FRt1/detection_loss‚0.5551704697531173 /loss/FBt2/detection_loss‚1.4167072668119713 % _timestamp‚1.7842742361713567e+09 epoch‚4 0loss/FRt2/assignment_loss‚1.4022398480891358‡éˆr“• ‚(:ˆ ǺçÒâÃñ¢ gpu.0.gpu‚45.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËIp– ‚(:ˆ ǺçÒÁ¬ø¦ disk.sda3.out‚ 40.32421875 disk.dm-0.in‚ 112.3046875 disk.rbd2.in‚0 disk.dm-1.out‚ 40.32421875# proc.memory.rssMB‚ 2776.18359375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 112.3046875 disk.rbd3.out‚0 disk.rbd2.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚545.8296203613281 disk.sda.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚6573.2080078125( proc.memory.availableMB‚ 740.72265625 network.sent‚ 2.544442e+06* disk./.usagePercent‚36.631083804004895 disk.rbd0.out‚0& disk.nvme0c0n1.out‚6573.2080078125 disk.sda3.in‚0$ memory_percent‚95.47898769378662 disk.sda1.in‚0 disk.sda.out‚ 40.32421875 disk.rbd1.out‚ 3.83203125! disk.dm-0.out‚6573.2080078125 disk.nvme0n1.in‚ 112.3046875 disk.drbd2165.in‚0* proc.memory.percent‚16.944479942321777 cpu‚25.935903117244756 network.recv‚419062 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.drbd2165.out‚0l£:6“— ‚(:ˆ ÖºçÒµ¶í¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðµ(˜ ‚(:Œ ÖºçÒôÿ®§ disk.sda.in‚0 disk.dm-1.in‚0* disk./.usagePercent‚36.631069211575436# disk./.usageGB‚545.829402923584 disk.rbd2.in‚0% disk.nvme0n1.out‚6581.89111328125 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 113.01953125 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 113.01953125' disk.nvme0c0n1.out‚6581.89111328125 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.83203125 disk.rbd3.out‚0" disk.dm-0.out‚6581.89111328125 network.sent‚ 2.594346e+06 network.recv‚423191 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 40.55078125$ memory_percent‚95.47867774963379 disk.rbd2.out‚0 disk.dm-1.out‚ 40.55078125 disk.nvme0n1.in‚ 113.01953125 disk.sda1.out‚0# proc.memory.rssMB‚ 2776.18359375* proc.memory.percent‚16.944479942321777 cpu‚25.940027728476093 disk.sda.out‚ 40.55078125' proc.memory.availableMB‚ 740.77343754;®´2™ - _wandbruntime‚1468  _runtime‚1468ñ‰àÅš ¿ _step‚136 & lr-AdamW/pg1‚0.0004675639538532806 & lr-AdamW/pg2‚0.0004675639538532806 trainer/global_step‚1699 # _timestamp‚1.78427427587506e+09 _runtime‚1468.000605009 “Ê› Ä & lr-AdamW/pg1‚0.0004675639538532806 & lr-AdamW/pg2‚0.0004675639538532806 trainer/global_step‚1699 # _timestamp‚1.78427427587506e+09 _runtime‚1468.000605009 _step‚136ˆ¨¢k2œ - _wandbruntime‚1468  _runtime‚1468_«|ƒp ÔO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258530491912198 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15739428 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58832803577344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795170866176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258530849501448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258063857958912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13460926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚1699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6640789 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚1468.001845411 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259859370164224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15729451 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3658076 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57016516773888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256738061815808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6640534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256738078855168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1816393992704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259859011785728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258063857958912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1816305415680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842742758760796e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15729451 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13460924 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1792770646280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1792430096390 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13460924 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 WJDeviceStatsMonitor.on_train_bSWõ¤jatch_start/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1795170866176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57016498161664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3658328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2982458 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259859370164224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259859011785728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258063840919552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚137 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0*‚)Àß'ž ÙO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3658076 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57016516773888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256738061815808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6640534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256738078855168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1816393992704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259859011785728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258063857958912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1816305415680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842742758760796e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15729451 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13460924 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1792770646280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1792430096390 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13460924 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1795170866176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57016498161664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3658328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2982458 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259859370164224 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259859011785728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258063840919552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚137 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258530491912198 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15739428 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58832803577344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795170866176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258530849501448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258063857958912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13460926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚1699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6640789 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚1468.001845411 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259859370164224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15729451 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968‰ÜÕœ!2Ÿ - _wandbruntime‚1468  _runtime‚1468^o…£ã&  ÝM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3660392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58867264522240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256884589527040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256884572487680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3660136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784274276487594e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57049945532416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29207059 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1793781700114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13468558 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468556 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258211081949184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15738503 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6644598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260007265442304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58867365412864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817419880448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260007623827968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258211098988544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚1468.613434763 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15748494 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260007623827968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1796183493120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13468556 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2984203 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1796524839424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15738503 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6644339 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚138 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539#;Éè&¡ âM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15748494 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260007623827968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1796183493120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13468556 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2984203 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1796524839424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15738503 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6644339 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚138 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3660392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58867264522240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256884589527040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256884572487680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3660136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784274276487594e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57049945532416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29207059 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1793781700114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13468558 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468556 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258211081949184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15738503 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6644598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260007265442304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58867365412864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817419880448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260007623827968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258211098988544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚1468.613434763Š˜Ca-2¢ - _wandbruntime‚1468  _runtime‚1468½É W£ € trainer/global_step‚1699 % _timestamp‚1.7842742764909341e+09 _runtime‚1468.616281047 0loss/FRt2/assignment_loss‚1.1736097273510924 2loss/SRqqt2/assignment_loss‚0.6354534540596077 1loss/FBt1/assignment_loss‚0.13197513198455557 0loss/SRqqt2/detection_loss‚0.680621545222338 0loss/FBt1/detection_loss‚0.4751+LY¯78290673097395 0loss/FRt1/assignment_loss‚2.4752642423607583 0loss/SRqqt1/detection_loss‚1.287115507795439 _step‚139 2loss/SRqqt1/assignment_loss‚0.7764248055218441 /loss/FRt1/detection_loss‚0.6076224571122217 /loss/FRt2/detection_loss‚1.3078449084577621 /loss/FBt2/detection_loss‚1.2177650421857833 1loss/FBt2/assignment_loss‚0.13146125138931256 %loss/total_loss‚10.90094098017169 epoch‚4Ñš^þ ¤ … 1loss/FBt2/assignment_loss‚0.13146125138931256 %loss/total_loss‚10.90094098017169 epoch‚4 trainer/global_step‚1699 % _timestamp‚1.7842742764909341e+09 _runtime‚1468.616281047 0loss/FRt2/assignment_loss‚1.1736097273510924 2loss/SRqqt2/assignment_loss‚0.6354534540596077 1loss/FBt1/assignment_loss‚0.13197513198455557 0loss/SRqqt2/detection_loss‚0.680621545222338 0loss/FBt1/detection_loss‚0.47578290673097395 0loss/FRt1/assignment_loss‚2.4752642423607583 0loss/SRqqt1/detection_loss‚1.287115507795439 _step‚139 2loss/SRqqt1/assignment_loss‚0.7764248055218441 /loss/FRt1/detection_loss‚0.6076224571122217 /loss/FRt2/detection_loss‚1.3078449084577621 /loss/FBt2/detection_loss‚1.2177650421857833‹ë4Ø“¥ ‚(:ˆ åºçÒ諧¤ gpu.0.gpu‚45.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚228.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.7372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pîë¡ ¦ ‚(:€ åºçÒÀôΦ disk.dm-1.in‚0 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.out‚ 40.703125$ memory_percent‚95.47560214996338 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 113.01953125 disk.sda.out‚ 40.703125 disk.rbd3.in‚0 disk.dm-0.in‚ 113.01953125 disk.dm-0.out‚6585.322265625 cpu‚25.944606841021113 network.recv‚426506) disk./.usagePercent‚36.63116367835558# disk./.usageGB‚545.830810546875# disk.nvme0n1.out‚6585.322265625* proc.memory.percent‚16.939997673034668 disk.sda1.in‚0( proc.memory.availableMB‚ 741.27734375 disk.sda1.out‚0% disk.nvme0c0n1.out‚6585.322265625 disk.sda2.in‚0 disk.sda3.in‚0 network.sent‚ 2.612433e+06 disk.rbd2.in‚0 disk.nvme0n1.in‚ 113.01953125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.89453125# proc.memory.rssMB‚ 2775.44921875 disk.drbd2165.out‚0 disk.dm-1.out‚ 40.703125 disk.rbd3.out‚0 proc.cpu.threads‚17èÖû“§ ‚(:ˆ ôºçÒïüî¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëëAr¨ ‚(:ý ôºçÒ¯Þž§ network.recv‚430609 disk.sda3.out‚ 40.953125 disk.dm-1.out‚ 40.953125 disk.rbd0.in‚0* proc.memory.percent‚16.944479942321777 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 network.sent‚ 2.662342e+06* disk./.usagePercent‚36.630831380576126$ disk./.usageGB‚545.8258590698242 disk.dm-1.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2776.18359375" disk.nvme0c0n1.in‚ 113.84765625 disk.sda.out‚ 40.953125 disk.rbd1.out‚ 3.89453125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 113.84765625 disk.dm-0.out‚ 6594.94140625 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# memory_percent‚96.9810962677002 disk.sda.in‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 113.84765625$ disk.nvme0c0n1.out‚ 6594.94140625' proc.memory.availableMB‚ 494.6171875 cpu‚25.948884394345836" disk.nvme0n1.out‚ 6594.94140625 disk.sda2.out‚0B"þÀž© ‚(:“ ƒ»çÒ®’Ó£ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚263.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.790800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R6ˇª ‚(:ø ƒ»çÒë´Û§ network.sent‚ 2.668561e+06 disk.sda1.out‚0 disk.nvme0c0n1.in‚114.5625 disk.rbd1.in‚0) disk./.usagePercent‚36.63085749334462" disk.dm-0.out‚6601.95361328125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6601.95361328125 disk.sda.out‚41.40625 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 4.03515625$ disk./.usageGB‚545.8262481689453 disk.rbd3.in‚0 disk.dm-0.in‚114.5625 disk.sda.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0$ memory_percent‚96.97654247283936 cpu‚25.952113421658467 disk.rbd2.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚41.40625# proc.memory.rssMB‚ 2775.44921875 network.recv‚433148% disk.nvme0n1.out‚6601.95361328125 disk.rbd0.out‚0( proc.memory.availableMB‚ 495.36328125* proc.memory.percent‚16.939997673034668 disk.nvme0n1.in‚114.5625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚41.40625"­Sµ2« - _wandbruntime‚1508  _runtime‚1508Жñ?Ǭ Á & lr-AdamW/pg1‚0.0004671650152327698 & lr-AdamW/pg2‚0.0004671650152327698 trainer/global_step‚1749 % _timestamp‚1.7842743163757553e+09 _runtime‚1508.501402966 _step‚140¿î‰Ì­ Æ & lr-AdamW/pg1‚0.0004671650152327698 & lr-AdamW/pg2‚0.0004671650152327698 trainer/global_step‚1749 % _timestamp‚1.7842743163757553e+09 _runtime‚1508.501402966 _step‚140Œ¾4øj2® - _wandbruntime‚1508  _runtime‚1508yÄè=Ú'¯ ÔO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1846207707136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1867563070976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843387133188 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6831525 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16183903 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1845866368000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843046583298 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1867649550848 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265425909440512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267272117147648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13842524 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264063612448768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13842526 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6831259 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3761551 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13842524 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1845866368000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3761814 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3069711 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267271758769152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265425892401152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60555499159040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58687936088064 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265906999581956 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7842743163769214e+09 _runtime‚1508.502834428 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60555604251136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚141 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264063595409408 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265425892401152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1846207707136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58687954700288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267272117147648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13842526 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16193880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3069708 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267271758769152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265425909440512éÁ=¥ß'° ÙO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16183903 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1845866368000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843046583298 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1867649550848 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265425909440512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267272117147648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13842524 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264063612448768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13842526 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6831259 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3761551 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13842524 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1845866368000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3761814 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3069711 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267271758769152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265425892401152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60555499159040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58687936088064 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265906999581956 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7842743163769214e+09 _runtime‚1508.502834428 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60555604251136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚141 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264063595409408 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265425892401152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1846207707136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58687954700288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267272117147648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13842526 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16193880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3069708 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267271758769152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265425909440512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1846207707136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1867563070976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1843387133188 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6831525 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616öOª[2± - _wandbruntime‚1509  _runtime‚1509ÚJâ² ÞM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1844057637020 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267420370811392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3763920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58721364846592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 trainer/global_step‚1749 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60589958426624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58721383458816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16202946 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3763663 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚142 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265573133430784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1844398187022 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3071453 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264210106081280 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13850156 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13850158 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266054521307662 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3071456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264210123120640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265573150471gŠ¶Ê0144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13850158 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚265573133430784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265573150470144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1846878994944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1847220341248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16192955 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266054163718300 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6835376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6835116 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1868593580032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0nôé&³ ãM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚265573133430784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265573150470144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1846878994944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1847220341248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16192955 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266054163718300 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6835376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6835116 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1868593580032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1844057637020 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267420370811392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3763920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58721364846592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 trainer/global_step‚1749 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60589958426624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58721383458816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16202946 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3763663 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚142 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265573133430784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1844398187022 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3071453 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264210106081280 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13850156 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13850158 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266054521307662 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3071456 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264210123120640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265573150470144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13850158 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591ŽjÖ„2´ - _wandbruntime‚1509  _runtime‚15096KÙmµ ‚ trainer/global_step‚1749 _runtime‚ 1509.11837803 _step‚143 0loss/FBt1/detection_loss‚0.43114858935062644 0loss/FRt2/assignment_loss‚1.2269880248552887 2loss/SRqqt2/assignment_loss‚0.6549202585996459 /loss/FRt2/detection_loss‚1.3886164421502416 &loss/total_loss‚11.030707178725189 % _timestamp‚1.7842743169929318e+09 1loss/SRqqt2/detection_loss‚0.6800745282350144 epoch‚4 0loss/FRt1/assignment_loss‚2.3520045849794897 2loss/SRqqt1/assignment_loss‚0.6861572473131324 1loss/FBt1/assignment_loss‚0.09362718467230825 1loss/FBt2/assignment_loss‚0.17125369605391388 /loss/FRt1/detection_loss‚0.5905855457471013 1loss/SRqqt1/detection_loss‚1.3500182328907322 /loss/FBt2/detection_loss‚1.4053128438776936j…È9 ¶ ‡ 0loss/FBt1/detection_loss‚0.43114858935062644 0loss/FRt2/assignment_loss‚1.2269880248552887 2loss/SRqqt2/assignment_loss‚0.6549202585996459 /loss/FRt2/detection_loss‚1.3886164421502416 &loss/total_loss‚11.030707178725189 % _timestamp‚1.7842743169929318e+09 1loss/SRqqt2/detection_loss‚0.6800745282350144 epoch‚4 0loss/FRt1/assignment_loss‚2.3520045849794897 2loss/SRqqt1/assignment_loss‚0.6861572473131324 1loss/FBt1/assignment_loss‚0.09362718467230825 1loss/FBt2/assignment_loss‚0.17125369605391388 /loss/FRt1/detection_loss‚0.5905855457471013 1loss/SRqqt1/detection_loss‚1.3500182328907322 /loss/FBt2/detection_loss‚1.4053128438776936 trainer/global_step‚1749 _runtime‚ 1509.11837803 _step‚143=‹~u“· ‚(:ˆ ’»çÒÞªú¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅoÉ° ¸ ‚(:‚ ’»çÒ¯µ†¨ disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 41.62109375 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 115.390625$ memory_percent‚96.98095321655273 disk.sda.in‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚6613.3349609375 cpu‚25.956406706021397) disk./.usagePercent‚36.63091381500216 disk.dm-1.in‚0 network.recv‚436416 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 4.03515625 disk.rbd2.out‚0 disk.drbd2165.in‚0 network.sent‚ 2.684567e+06 disk.sda3.in‚0! disk.dm-0.out‚6613.3349609375 disk.sda1.in‚0# proc.memory.rssMB‚ 2776.18359375 disk.rbd3.out‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 494.640625 disk.sda3.out‚ 41.62109375 disk.dm-1.out‚ 41.62109375 disk.rbd0.in‚0* proc.memory.percent‚16.944479942321777$ disk./.usageGB‚545.8270874023438 disk.nvme0n1.in‚ 115.390625$ disk.nvme0n1.out‚6613.3349609375 disk.nvme0c0n1.in‚ 115.390625 proc.cpu.threads‚17bÙБ¹ ‚(:† ¡»çÒíßÕ£ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0— a\º ‚(:ü ¡»çÒàšþ¤$ disk./.usageGB‚545.8274688720703 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.959783677165692 disk.sda.in‚0 disk.sda.out‚ 41.8828125 disk.dm-1.in‚0 disk.dm-1.out‚ 41.8828125# proc.memory.rssMB‚ 2775.44921875 disk.nvme0c0n1.in‚ 115.390625 disk.rbd1.out‚ 4.09765625 disk.sda1.out‚0* disk./.usagePercent‚36.630939415755584 disk.rbd2.out‚0 disk.nvme0n1.in‚ 115.390625# disk.nvme0n1.out‚6616.822265625 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚0# memory_percent‚96.9759464263916 disk.sda1.in‚0 network.sent‚ 2.73654e+06 disk.rbd2.in‚0 disk.sda3.out‚ 41.8828125' proc.memory.availableMB‚ 495.4609375 disk.dm-0.in‚ 115.390625 disk.rbd0.in‚0% disk.nvme0c0n1.out‚6616.822265625 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-0.out‚6616.822265625 disk.rbd0.out‚0 network.recv‚441009 disk.rbd3.in‚0* proc.memory.percent‚16.939997673034668Õ%'¯“» ‚(:ˆ °»çÒ€Ö€¤ gpu.0.gpu‚66.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:žÉ¼ ‚(:ˆ °»çÒ…¯Â¦ disk.sda2.in‚0 disk.rbd2.out‚0' disk.nvme0c0n1.out‚6628.42041015625* proc.memory.percent‚16.944479942321777 network.sent‚ 2.739605e+06 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 42.1640625 disk.sda3.out‚ 42.1640625* disk./.usagePercent‚36.630939415755584 disk.rbd2.in‚0 proc.cpu.threads‚17 network.recv‚444549 disk.rbd1.in‚0 disk.rbd1.out‚ 4.09765625 disk.dm-0.in‚ 116.10546875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 116.10546875 disk.sda.in‚0$ memory_percent‚96.97895050048828 disk.dm-1.in‚0" disk.dm-0.out‚6628.42041015625 disk.rbd0.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2776.18359375 cpu‚25.963725904319656 disk.dm-1.out‚ 42.1640625 disk.sda3.in‚0% proc.memory.availableMB‚ 494.96875$ disk./.usageGB‚545.8274688720703 disk.rbd3.out‚0 disk.nvme0n1.in‚ 116.10546875% disk.nvme0n1.out‚6628.42041015625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0^KÎ2½ - _wandbruntime‚1548  _runtime‚1548^Yã°Ç¾ Á _step‚144 & lr-AdamW/pg1‚0.0004667512507743636 & lr-AdamW/pg2‚0.0004667512507743636 trainer/global_step‚1799 % _timestamp‚1.7842743566668024e+09 _runtime‚1548.7965387328JoÿÌ¿ Æ trainer/global_step‚1799 % _timestamp‚1.7842743566668024e+09 _runtime‚1548.796538732 _step‚144 & lr-AdamW/pg1‚0.0004667512507743636 & lr-AdamW/pg2‚0.0004667512507743636{³ÚÍ2À - _wandbruntime‚1548  _runtime‚1548“Ô·Ú'Á ÔO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16638355 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272787960922112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1896561869824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3864732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62278174523392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7021690 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1918886988800 trainer/global_step‚1799 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272787943882752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3864982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1896561869824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918800508928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271389146042368 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7021943 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274684505752576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3156958 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16648332 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14224126 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16648332 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60359392626688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271389129003008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842743566675632e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274684864131072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚274684864131072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14224124 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1893663070206 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14224126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272787960922112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274684505752576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14224124 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1896903208960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1894003620096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088xíþà ÙO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30862479 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1896903208960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚1548.796712179 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7021690 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1918886988800 trainer/global_step‚1799 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272787943882752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3864982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1896561869824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918800508928 `LDeviceStatsMonitor.on_train_batch_start/reqE²ýÎÿuested_bytes.large_pool.allocated‚271389146042368 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7021943 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274684505752576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3156958 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16648332 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14224126 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16648332 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60359392626688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271389129003008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842743566675632e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274684864131072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚274684864131072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14224124 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1893663070206 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14224126 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272787960922112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274684505752576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14224124 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1896903208960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1894003620096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16638355 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272787960922112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1896561869824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3864732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62278174523392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231758 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14231756 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919835847168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274833117794816 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842743572799957e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1897574496768 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62312638620160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60392821385216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1895014673930 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231758 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3866964 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚1799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30889156 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272935201951744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1897915843072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60392802772992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62312737413632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919916028416 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16647407 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273430671388170 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚146 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274833117794816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16657398 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16647407 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7025413 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1894674123928 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3866710 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16657398 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272935201951744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14231756 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271535656714240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842743572799957e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1897574496768 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62312638620160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60392821385216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1895014673930 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231758 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3866964 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚1799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30889156 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272935201951744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1897915843072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60392802772992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62312737413632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919916028416 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16647407 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273430671388170 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚146 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274833117794816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16657398 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16647407 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7025413 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1894674123928 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3866710 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16657398 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272935201951744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14231756 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271535656714240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14231758 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14231756 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919835847168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274833117794816 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282097252736000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282097611114496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 trainer/global_step‚1849 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3968695 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000867754496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280659299742972 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842743971609724e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282097252736000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14605724 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1970037201408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17102784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280149995364352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278714679635968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1589.286760699 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282097611114496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7212627 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1947598710784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1947598710784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3968419 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969948624384 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1947257371648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17102784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280150012403712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278714662596608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14605724 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17092807 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280149995364352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31698531 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62030811940864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280658942153722 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282097611114496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7212627 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1947598710784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1947598710784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3968419 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969948624384 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1947257371648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17102784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280150012403712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278714662596608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14605724 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17092807 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280149995364352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31698531 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62030811940864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280658942153722 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282097252736000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282097611114496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 trainer/global_step‚1849 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3968695 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000867754496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280659299742972 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842743971609724e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282097252736000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14605724 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1970037201408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088•cQ"–2Ó - _wandbruntime‚1589  _runtime‚1589MÊÅ7ä&Ô ÞM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17101859 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14613358 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62064240699392 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278861190307840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278861173268480 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280297236393984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14613358 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64035316582400 trainer/global_step‚1849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚282245506392576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280806463879316 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3970726 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7216682 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280806821468678 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3245956 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64035209400320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17111850 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1948611344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14613356 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14613356 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970968700928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3970476 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282245864778240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1945290610836 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1948269998592 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7216429 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚150 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3245953 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17101859 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62064259311616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282245864778240 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360î¶)¬Õ ãM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7216682 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280806821468678 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3245956 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64035209400320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17111850 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1948611344896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14613356 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14613356 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970968700928 ;5De•Ð€}ÏviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3970476 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282245864778240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1945290610836 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1948269998592 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7216429 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚150 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3245953 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17101859 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62064259311616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282245864778240 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17101859 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14613358 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62064240699392 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278861190307840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278861173268480 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280297236393984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14613358 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64035316582400 trainer/global_step‚1849 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚282245506392576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280806463879316 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3970726 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0–ÿȉI2Ö - _wandbruntime‚1589  _runtime‚1589o× ü /loss/FRt1/assignment_loss‚2.278314261390759 1loss/SRqqt2/assignment_loss‚0.649907986896688 /loss/FBt2/assignment_loss‚0.230304716414239 epoch‚4 _runtime‚1589.902110285 0loss/FRt2/assignment_loss‚1.1268697804617651 1loss/SRqqt1/assignment_loss‚0.722212000759998 /loss/FRt1/detection_loss‚0.5702616648623423 /loss/FRt2/detection_loss‚1.4049527393129555 _step‚151 1loss/FBt1/assignment_loss‚0.08945351078845817 /loss/FBt1/detection_loss‚0.5093624546412671 /loss/FBt2/detection_loss‚1.1298899778723717 %loss/total_loss‚10.68174762175613 trainer/global_step‚1849 % _timestamp‚1.7842743977764306e+09 1loss/SRqqt1/detection_loss‚1.2503677483791957 1loss/SRqqt2/detection_loss‚0.7198507799760918 ¦B Ø  1loss/SRqqt1/assignment_loss‚0.722212000759998 /loss/FRt1/detection_loss‚0.5702616648623423 /loss/FRt2/detection_loss‚1.4049527393129555 _step‚151 1loss/FBt1/assignment_loss‚0.08945351078845817 /loss/FBt1/detection_loss‚0.5093624546412671 /loss/FBt2/detection_loss‚1.1298899778723717 %loss/total_loss‚10.68174762175613 trainer/global_step‚1849 % _timestamp‚1.7842743977764306e+09 1loss/SRqqt1/detection_loss‚1.2503677483791957 1loss/SRqqt2/detection_loss‚0.7198507799760918 /loss/FRt1/assignment_loss‚2.278314261390759 1loss/SRqqt2/assignment_loss‚0.649907986896688 /loss/FBt2/assignment_loss‚0.230304716414239 epoch‚4 _runtime‚1589.902110285 0loss/FRt2/assignment_loss‚1.1268697804617651—㉓٠‚(:ˆ Ý»çÒ€ù¦¥ gpu.0.gpu‚66.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÄË*Ú ‚(:‰ Ý»çÒÐô±§ disk.sda.out‚ 43.55078125 disk.dm-1.out‚ 43.55078125 disk.rbd1.out‚ 4.23828125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 118.4765625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2775.43359375) disk./.usagePercent‚36.63141200566379$ disk./.usageGB‚545.8345108032227 disk.sda3.in‚0% disk.nvme0n1.out‚6657.96337890625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6657.96337890625 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 118.4765625 disk.sda.in‚0 disk.sda3.out‚ 43.55078125 disk.rbd1.in‚0 disk.rbd2.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 495.29296875 proc.cpu.threads‚17 network.sent‚ 2.812728e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.97697162628174* proc.memory.percent‚16.939902305603027 cpu‚25.97417176936373 network.recv‚456772 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚6657.96337890625! disk.nvme0c0n1.in‚ 118.4765625"r¯‚“Û ‚(:ˆ ì»çÒœ³¤ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³”ØÜ ‚(:ˆ ì»çÒ¶Üæ¦ disk.rbd3.out‚0 disk.rbd2.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6671.38525390625# proc.memory.rssMB‚ 2776.18359375* proc.memory.percent‚16.944479942321777$ disk./.usageGB‚545.8838386535645 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 4.23828125 disk.rbd3.in‚0 disk.sda1.out‚0$ memory_percent‚96.98119163513184 disk.sda2.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 118.4765625 network.sent‚ 2.877965e+06 disk.sda3.in‚0 disk.sda3.out‚ 45.12890625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 118.4765625' proc.memory.availableMB‚ 494.6015625 disk.rbd0.out‚0) disk./.usagePercent‚36.63472243908905 disk.rbd1.in‚0 disk.dm-0.in‚ 118.4765625" disk.dm-0.out‚6671.38525390625 disk.rbd2.out‚0% disk.nvme0n1.out‚6671.38525390625 proc.cpu.threads‚17 network.recv‚461354 disk.drbd2165.in‚0 cpu‚25.97775832759949 disk.sda.out‚ 45.12890625 disk.dm-1.out‚ 45.12890625 disk.sda1.in‚0}ÿWÚ’Ý ‚(:‡ û»çÒƒ­ù¤ gpu.0.gpu‚18.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{G°Þ ‚(:Œ û»çÒԳΩ disk.sda.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 119.19140625 disk.nvme0n1.in‚ 119.19140625$ disk./.usageGB‚545.8299598693848 disk.sda.out‚ 45.79296875 disk.dm-1.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚6680.79052734375 disk.sda1.in‚0* proc.memory.percent‚16.939902305603027* disk./.usagePercent‚36.631106588675436 disk.dm-0.in‚ 119.19140625 disk.drbd2165.out‚0# memory_percent‚96.9773530960083( proc.memory.availableMB‚ 495.23046875# proc.memory.rssMB‚ 2775.43359375 network.sent‚ 2.883098e+06 disk.sda3.out‚ 45.79296875 disk.rbd2.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚6680.79052734375 disk.rbd1.out‚ 4.31640625 disk.rbd0.in‚0 network.recv‚464794 disk.sda2.out‚0 disk.dm-1.out‚ 45.79296875 disk.rbd1.in‚0" disk.dm-0.out‚6680.79052734375 disk.rbd2.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 cpu‚25.98041845535737 proc.cpu.threads‚17jyÑg2ß - _wandbruntime‚1629  _runtime‚1629»Ï©Éà à _step‚152 ' lr-AdamW/pg1‚0.00046587935159566593 ' lr-AdamW/pg2‚0.00046587935159566593 trainer/global_step‚1899 % _timestamp‚1.7842744376067464e+09 _runtime‚1629.732353258Ž]œ´Îá È ' lr-AdamW/pg1‚0.00046587935159566593 ' lr-AdamW/pg2‚0.00046587935159566593 trainer/global_step‚1899 % _timestamp‚1.7842744376067464e+09 _runtime‚1629.732353258 _step‚152˜.ï,k2â - _wandbruntime‚1629  _runtime‚16297Šý|Ú'ã ÔO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32534583 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14987326 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4072214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17557236 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287512046845952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287512046845952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7403675 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288035449823480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14987326 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286040196190208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289510358097920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65723350175744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1994896044022 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14987324 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1998294212608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7842744376078022e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2021081696256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1997952873472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1995236593912 _runtime‚1629.736434083 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65723238792192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1998294212608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289510358097920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17547259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287512063885312 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7403391 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚1899 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3331458 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4071933 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63702249867264 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289509999719424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32544562 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17547259 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14987324 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2020988924928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289509999719424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088Æ·»ä ÙO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1998294212608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289510358097920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17547259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287512063885312 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7403391 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚1899 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3331458 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4071933 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63702249867264 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289509999719424 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32544562 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17547259 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14987324 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2020988924928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289509999719424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32534583 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14987326 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4072214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17557236 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287512046845952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287512046845952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7403675 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288035449823480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14987326 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286040196190208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289510358097920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65723350175744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1994896044022 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14987324 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1998294212608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7842744376078022e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2021081696256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1997952873472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1995236593912 _runtime‚1629.736434083 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65723238792192™ZÉçv2å - _wandbruntime‚1630  _runtime‚1630ÏH oã&æ ÝM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63735678625792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286186723901440 % _timestamp‚1.7842744382228286e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚154 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17556311 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287659287875584 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7407198 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2022098140672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289658253376000 trainer/global_step‚1899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14994956 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2022009570816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14994958 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4074262 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1995907097744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1998965500416 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17566302 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1999306846720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289658611761664 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65757795378688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288182613959824žî#è&ç âM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289658611761664 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65757795378688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288182613959824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63735678625792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286186723901440 % _timestamp‚1.7842744382228286e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚154 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17556311 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287659287875584 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7407198 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2022098140672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289658253376000 trainer/global_step‚1899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14994956 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2022009570816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14994958 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4074262 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1995907097744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1998965500416 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17566302 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1999306846720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2šØ=«2è - _wandbruntime‚1630  _runtime‚1630UÄgé € 1loss/FBt1/assignment_loss‚0.10257460248165896 epoch‚4 /loss/FRt1/assignment_loss‚2.323780009176308 .loss/FRt2/detection_loss‚1.335116490555187 /loss/FBt1/detection_loss‚0.4461453056500222 0loss/FRt2/assignment_loss‚1.2762543675090585 2loss/SRqqt1/assignment_loss‚0.8817416596443456 /loss/FRt1/detection_loss‚0.5284416893162857 1loss/SRqqt2/detection_loss‚0.6553822055899481 /loss/FBt2/detection_loss‚1.2984598905399993 trainer/global_step‚1899 2loss/FBt2/assignment_loss‚0.013887758558633289 1loss/SRqqt1/detection_loss‚1.2198383971098885 %loss/total_loss‚10.76221583908032 % _timestamp‚1.7842744382244651e+09 _runtime‚1630.349792627 _step‚155 2loss/SRqqt2/assignment_loss‚0.6805934629489842¹#t ê … 0loss/FRt2/assignment_loss‚1.2762543675090585 2loss/SRqqt1/assignment_loss‚0.8817416596443456 /loss/FRt1/detection_loss‚0.5284416893162857 1loss/SRqqt2/detection_loss‚0.6553822055899481 /loss/FBt2/detection_loss‚1.2984598905399993 trainer/global_step‚1899 2loss/FBt2/assignment_loss‚0.013887758558633289 1loss/SRqqt1/detection_loss‚1.2198383971098885 %loss/total_loss‚10.76221583908032 % _timestamp‚1.7842744382244651e+09 _runtime‚1630.349792627 _step‚155 2loss/SRqqt2/assignment_loss‚0.6805934629489842 1loss/FBt1/assignment_loss‚0.10257460248165896 epoch‚4 /loss/FRt1/assignment_loss‚2.323780009176308 .loss/FRt2/detection_loss‚1.335116490555187 /loss/FBt1/detection_loss‚0.4461453056500222›ÛâE±žë ‚(:“ Š¼çÒÃÍ´¤ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³&+ì ‚(: Š¼çÒ훾¦ proc.cpu.threads‚17 disk.sda2.out‚0( proc.memory.availableMB‚ 494.49609375 disk.rbd1.out‚ 4.31640625 disk.sda1.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6695.83349609375 disk.sda3.in‚0$ memory_percent‚96.98183536529541 disk.dm-1.in‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 120.01953125 disk.sda2.in‚0 network.sent‚ 2.886062e+06* disk./.usagePercent‚36.631111708826126 disk.rbd3.in‚0% disk.nvme0n1.out‚6695.83349609375 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 47.95703125# proc.memory.rssMB‚ 2776.18359375 network.recv‚468160 disk.dm-0.in‚ 120.01953125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 47.95703125* proc.memory.percent‚16.944479942321777 cpu‚25.98367003132923 disk.dm-1.out‚ 47.95703125 disk.rbd2.out‚0 disk.sda.in‚0$ disk./.usageGB‚545.8300361633301 disk.rbd1.in‚0" disk.dm-0.out‚6695.83349609375 disk.nvme0n1.in‚ 120.01953125P¼[}í ‚(:„ ™¼çÒؾ‚¥ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EH›Þýî ‚(:ò ™¼çÒºµ¤§ network.sent‚ 2.949171e+06" disk.dm-0.out‚6702.65576171875 disk.rbd2.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.631110684795985 disk.dm-0.in‚ 120.734375# disk./.usageGB‚545.830020904541 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚6702.65576171875# proc.memory.rssMB‚ 2775.43359375 disk.rbd1.out‚ 4.44140625 disk.rbd2.in‚0% disk.nvme0n1.out‚6702.65576171875 disk.sda3.in‚0$ memory_percent‚96.97692394256592 network.recv‚472807 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 120.734375 disk.sda.out‚48.5 disk.dm-1.out‚48.5 disk.sda3.out‚48.5 disk.rbd1.in‚0 disk.nvme0n1.in‚ 120.734375 disk.sda.in‚0 cpu‚25.987621919344456 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 495.30078125 disk.rbd3.in‚0* proc.memory.percent‚16.939902305603027²ð&“ï ‚(:ˆ ¨¼çÒÁ‘ä¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G,ð ‚(:ö ¨¼çÒÒý‹§" disk.nvme0n1.out‚ 6716.79296875 disk.sda3.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.956352e+06 disk.nvme0c0n1.in‚121.5625 disk.sda.in‚0 disk.dm-1.out‚ 50.04296875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚121.5625 disk.sda1.out‚0 disk.sda.out‚ 50.04296875$ memory_percent‚96.98224067687988* proc.memory.percent‚16.944479942321777$ disk./.usageGB‚545.8348617553711$ disk.nvme0c0n1.out‚ 6716.79296875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 6716.79296875# proc.memory.rssMB‚ 2776.18359375 disk.rbd0.out‚0 disk.sda3.out‚ 50.04296875 disk.rbd1.out‚ 4.44140625 disk.rbd3.in‚0' proc.memory.availableMB‚ 494.4296875 cpu‚25.98992899208924 disk.sda2.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚36.63143555835694 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.dm-0.in‚121.5625 network.recv‚476374b>È2ñ - _wandbruntime‚1670  _runtime‚1670/IøÄò ¾ trainer/global_step‚1949 # _timestamp‚1.78427447787471e+09 _runtime‚ 1670.00265678 _step‚156 & lr-AdamW/pg1‚0.0004654212728758294 & lr-AdamW/pg2‚0.0004654212728758294ƒýPÉó à & lr-AdamW/pg1‚0.0004654212728758294 & lr-AdamW/pg2‚0.0004654212728758294 trainer/global_step‚1949 # _timestamp‚1.78427447787471e+09 _runtime‚ 1670.00265678 _step‚156œ‹û•2ô - _wandbruntime‚1670  _runtime‚1670p¤Wõ ÔO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2072266800128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18001711 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18011688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293365746823168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296923105081344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2048989714432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2048648375296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3418708 trainer/global_step‚1949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚65373687793664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842744778789344e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15368926 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7594862 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294874098327552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15368926 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚1670.004639643 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296923105081344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2048989714432 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296922746702848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.cubwœ ƒrrent‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2045512530930 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3418711 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4175874 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1æ‘Y¿ß'ö ÙO KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296922746702848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2045512530930 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3418711 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4175874 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2072266800128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18001711 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18011688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293365746823168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296923105081344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2048989714432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2048648375296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3418708 trainer/global_step‚1949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚65373687793664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842744778789344e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15368926 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7594862 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294874098327552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15368926 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚1670.004639643 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296923105081344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2048989714432ÿŒ ]2÷ - _wandbruntime‚1670  _runtime‚1670>ÈÇ ä&ø ÞM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293512257495040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4177980 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67480313661952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295559121629694 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7598433 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2046864134654 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18010763 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295558764040332 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295021356396544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚1670.615294399 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293512240455680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33387319 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049661002240 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7598696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚65407116552192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295021339357184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15376556 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2073285679616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2073197109760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2046523584652 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7842744784894445e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049661002240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297071358745088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2050002348544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2050002348544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1949 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295021339357184 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293512257495040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4177980 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67480313661952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295559121629694 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7598433 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2046864134654 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18010763 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295558764040332 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295021356396544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚1670.615294399 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293512240455680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33387319 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049661002240 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7598696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚65407116552192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295021339357184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15376556 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.{#i„ peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2073285679616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2073197109760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2046523584652 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7842744784894445e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049661002240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297071358745088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2050002348544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2050002348544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1949 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295021339357184 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69168453190656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123327470592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304335493686272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18456163 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67045144332288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2099685216256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚1999 _runtime‚1710.552572462 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7786038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4280080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304335493686272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3505958 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302236149809152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302236166848512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3505961 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67045125720064 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2099343877120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ef«¡Þ'ˆ ØO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18466140 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300691263377408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2096469567728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15750526 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚161 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69168453190656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123327470592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304335493686272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18456163 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67045144332288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2099685216256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚1999 _runtime‚1710.552572462 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7786038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4280080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304335493686272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3505958 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302236149809152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302236166848512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3505961 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67045125720064 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2099343877120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18456163 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2099343877120 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚162 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304484105728512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18465215 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4282158 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18465215 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2100356504064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304483747342848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7789861 XFWÄúíDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2097140071560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302383390838784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67078573090816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302934914120840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100697850368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18475206 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124437030400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2097480621562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302383407878144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100356504064 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7790124 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4282418 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚1999 % _timestamp‚1.7842745190354874e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1711.161384386 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15758156 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67078554478592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300837791088640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300837774049280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169Ô§ºé&‹ ãM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300837791088640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300837774049280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304484105728512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15758158 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302935271710202 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3507706 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69203010121216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚162 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304484105728512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18465215 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4282158 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18465215 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2100356504064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304483747342848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7789861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2097140071560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302383390838784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67078573090816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302934914120840 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100697850368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18475206 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124437030400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2097480621562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302383407878144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100356504064 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7790124 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4282418 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚1999 % _timestamp‚1.7842745190354874e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1711.161384386 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15758156 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67078554478592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2¢g)2Œ - _wandbruntime‚1711  _runtime‚1711²Ž¬ ÿ 2loss/FBt2/assignment_loss‚0.014245889299974664 /loss/FRt1/detection_loss‚0.5692920258442004 /loss/FBt2/detection_loss‚1.2805010864422435 trainer/global_step‚1999 2loss/SRqqt2/assignment_loss‚0.6758522026735437 1loss/SRqqt2/detection_loss‚0.7056677191884387 # _timestamp‚1.78427451903808e+09 _runtime‚1711.163414949 _step‚163 0loss/FRt2/assignment_loss‚1.2758549240936337 1loss/FBt1/assignment_loss‚0.05468315738378554 -loss/FRt2/detection_loss‚1.35369280321971 /loss/FBt1/detection_loss‚0.7584101554621157 &loss/total_loss‚10.612404008206038 0loss/FRt1/assignment_loss‚2.2027770300749046 2loss/SRqqt1/assignment_loss‚0.7222990367291311 1loss/SRqqt1/detection_loss‚0.9991279777943562 epoch‚4õæj‹ Ž „ &loss/total_loss‚10.612404008206038 0loss/FRt1/assignment_loss‚2.2027770300749046 2loss/SRqqt1/assignment_loss‚0.7222990367291311 1loss/SRqqt1/detection_loss‚0.9991279777943562 epoch‚4 2loss/FBt2/assignment_loss‚0.014245889299974664 /loss/FRt1/detection_loss‚0.5692920258442004 /loss/FBt2/detection_loss‚1.2805010864422435 trainer/global_step‚1999 2loss/SRqqt2/assignment_loss‚0.6758522026735437 1loss/SRqqt2/detection_loss‚0.7056677191884387 # _timestamp‚1.78427451903808e+09 _runtime‚1711.163414949 _step‚163 0loss/FRt2/assignment_loss‚1.2758549240936337 1loss/FBt1/assignment_loss‚0.05468315738378554 -loss/FRt2/detection_loss‚1.35369280321971 /loss/FBt1/detection_loss‚0.7584101554621157£ìd¾“ ‚(:ˆ ä¼çÒݵƒ¥ gpu.0.gpu‚50.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚283.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À$¥Ý ‚(:ø ä¼çÒôâͦ% disk.nvme0c0n1.out‚6759.708984375# disk.nvme0n1.out‚6759.708984375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 491.26953125! disk.nvme0c0n1.in‚ 123.8203125 disk.sda2.in‚0 disk.rbd2.out‚0* proc.memory.percent‚16.964077949523926 disk.sda.in‚0 disk.nvme0n1.in‚ 123.8203125# proc.memory.rssMB‚ 2779.39453125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚54.03125 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 network.recv‚490745 disk.rbd1.out‚ 4.58203125 disk.dm-0.out‚6759.708984375( disk./.usagePercent‚36.6310861080727 disk.dm-0.in‚ 123.8203125 cpu‚26.02812670973663 disk.drbd2165.out‚0 network.sent‚ 3.045452e+06$ disk./.usageGB‚545.8296546936035 disk.sda.out‚54.03125 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0# memory_percent‚97.0015287399292 disk.dm-1.out‚54.03125 disk.rbd3.in‚0è±02‘ - _wandbruntime‚1728  _runtime‚1728$à²[â.’ †| particle/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9828827381134033 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12496083974838257 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.37716197967529297 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7922430634498596 *jet/accuracy_2_of_4‚0.7316775918006897 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6395388841629028 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15367400646209717 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922482371330261 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3634893000125885 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0198699701577425 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 /particle/accuracy_3_of_3‚0.9989922046661377 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.5997748374938965 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 % _timestamp‚1.7842745365836763e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4727853238582611 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9938073754310608 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6214054822921753 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5283333659172058 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06813240796327591 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08212827146053314 +jet/accuracy_1_of_1‚0.29684507846832275 _step‚164 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.982702374458313 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36354872584342957 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 trainer/global_step‚1999 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1667664349079132 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +jet/accuracy_2_of_2‚0.12397538870573044 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6724743843078613 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518519401550293 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9963316917419434 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 /particle/accuracy_4_of_4‚0.9432827830314636 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6736735701560974 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06813240796327591 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9926919341087341 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08207345008850098 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8287226557731628 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6450795531272888 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780851006507874 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15367400646209717 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5937976241111755 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7232629060745239 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26824361085891724 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7711504697799683 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5990688800811768 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9929200410842896 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6903077363967896 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7650444507598877 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 jet/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16182783246040344 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3685987591743469 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13710932433605194 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972379803657532 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.163332998752594 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08928634226322174 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18096888065338135 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6212652921676636 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9975515007972717 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007230561226606369 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083134651184 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.659759521484375 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6033086180686951 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6148968935012817 jet/accuracy_4_of_5‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6862655878067017 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7541194558143616 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09076927602291107 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7648558616638184 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18046113848686218 *jet/accuracy_1_of_3‚0.9125855565071106 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895849823951721 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5968447327613831 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9811508655548096 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.602264404296875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7431502342224121 6validation_average_jet_accuracy‚0.3416503369808197 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6684131026268005 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 epoch‚4 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1002209410071373 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6676768660545349 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14597661793231964 *jet/accuracy_1_of_4‚0.9581090807914734 ,jet/accuracy_4_of_4‚0.009931244887411594 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6960002779960632 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7351282238960266 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5885416269302368 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9956086277961731 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1542910784482956 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13606508076190948 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7922430634498596 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2384493052959442 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6544545292854309 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9208858013153076 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7216681241989136 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6856387257575989 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9911549687385559 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6248824596405029 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_3‚0.48035404086112976 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9885214567184448 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47181907296180725 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6704298853874207 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9851273894309998 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.06351815164089203 particle/accuracy_1_of_4‚1 *loss/val_total_loss‚10.500108899149694 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5372552275657654 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8509470820426941 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16793492436408997 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1495158076286316 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15882553160190582 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8375767469406128 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7459738850593567 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6163896322250366 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03033444471657276 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46773475408554077 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8122448921203613 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841141700744629 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941349029541016 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8087680339813232 ("Puk=:*rity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 _runtime‚1728.709899375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9106253385543823 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.46395039558410645 particle/accuracy_2_of_3‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731480598449707 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_4‚0.062254589051008224 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11580762267112732 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026308270171284676 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16055859625339508 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2751559615135193 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8571838140487671 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9963316917419434 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013331342488527298 jet/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14762365818023682 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3580140471458435 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11811677366495132 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16305291652679443 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518519401550293 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948374629020691 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9943203926086426 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7244325280189514 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01392628438770771 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5965863466262817 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.541997492313385 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9940476417541504 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1510065644979477 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22076694667339325 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08973367512226105 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16184952855110168 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7415293455123901 'particle/accuracy‚0.838681161403656 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6262717247009277 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37907737493515015 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9040180444717407 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7896825671195984 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10798285901546478 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6262717247009277 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6190804839134216 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7357553243637085 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1Êò¦>“ ‹| :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37907737493515015 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9040180444717407 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7896825671195984 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10798285901546478 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6262717247009277 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6190804839134216 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7357553243637085 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9828827381134033 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12496083974838257 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.37716197967529297 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7922430634498596 *jet/accuracy_2_of_4‚0.7316775918006897 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6395388841629028 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15367400646209717 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922482371330261 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3634893000125885 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0198699701577425 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 /particle/accuracy_3_of_3‚0.9989922046661377 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.5997748374938965 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 % _timestamp‚1.7842745365836763e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4727853238582611 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9938073754310608 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6214054822921753 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5283333659172058 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06813240796327591 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08212827146053314 +jet/accuracy_1_of_1‚0.29684507846832275 _step‚164 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.982702374458313 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36354872584342957 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 trainer/global_step‚1999 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1667664349079132 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +jet/accuracy_2_of_2‚0.12397538870573044 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6724743843078613 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.18518519401550293 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9963316917419434 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 /particle/accuracy_4_of_4‚0.9432827830314636 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6736735701560974 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06813240796327591 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9926919341087341 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08207345008850098 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8287226557731628 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6450795531272888 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780851006507874 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15367400646209717 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5937976241111755 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7232629060745239 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26824361085891724 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7711504697799683 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5990688800811768 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9929200410842896 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6903077363967896 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7650444507598877 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 jet/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16182783246040344 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3685987591743469 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13710932433605194 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972379803657532 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.163332998752594 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08928634226322174 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18096888065338135 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6212652921676636 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9975515007972717 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007230561226606369 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083134651184 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.659759521484375 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6033086180686951 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6148968935012817 jet/accuracy_4_of_5‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6862655878067017 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7541194558143616 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09076927602291107 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7648558616638184 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18046113848686218 *jet/accuracy_1_of_3‚0.9125855565071106 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895849823951721 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5968447327613831 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9811508655548096 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.602264404296875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7431502342224121 6validation_average_jet_accuracy‚0.3416503369808197 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6684131026268005 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 epoch‚4 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1002209410071373 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6676768660545349 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14597661793231964 *jet/accuracy_1_of_4‚0.9581090807914734 ,jet/accuracy_4_of_4‚0.009931244887411594 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6960002779960632 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7351282238960266 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5885416269302368 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9956086277961731 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1542910784482956 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13606508076190948 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7922430634498596 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2384493052959442 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6544545292854309 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9208858013153076 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7216681241989136 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6856387257575989 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9911549687385559 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6248824596405029 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_3‚0.48035404086112976 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9885214567184448 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47181907296180725 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6704298853874207 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9851273894309998 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.06351815164089203 particle/accuracy_1_of_4‚1 *loss/val_total_loss‚10.500108899149694 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5372552275657654 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8509470820426941 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16793492436408997 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1495158076286316 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15882553160190582 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8375767469406128 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7459738850593567 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6163896322250366 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03033444471657276 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46773475408554077 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8122448921203613 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841141700744629 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941349029541016 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8087680339813232 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 _runtime‚1728.709899375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9106253385543823 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.46395039558410645 particle/accuracy_2_of_3‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731480598449707 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_4‚0.062254589051008224 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11580762267112732 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026308270171284676 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16055859625339508 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2751559615135193 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8571838140487671 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9963316917419434 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013331342488527298 jet/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14762365818023682 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3580140471458435 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11811677366495132 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16305291652679443 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.18518519401550293 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948374629020691 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9943203926086426 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7244325280189514 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01392628438770771 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5965863466262817 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.541997492313385 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9940476417541504 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1510065644979477 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22076694667339325 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08973367512226105 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16184952855110168 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7415293455123901 'particle/accuracy‚0.838681161403656 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6262717247009277¤çy‘” ‚(:† ó¼çÒ½¬¥ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,´˜• ‚(:… ó¼çÒÀß²§ disk.rbd0.in‚0 network.recv‚495560$ disk./.usageGB‚545.8297157287598 disk.sda.in‚0 disk.sda3.out‚ 54.3515625 disk.dm-1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚7254.68896484375' proc.memory.availableMB‚ 744.3828125 disk.rbd1.out‚ 4.66796875 disk.rbd3.out‚0# proc.memory.rssMB‚ 2781.84765625 disk.sda2.out‚0 disk.rbd2.out‚0 disk.sda1.out‚0 cpu‚26.032346721739195 disk.dm-1.out‚ 54.3515625 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 123.8203125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 123.8203125 disk.nvme0n1.in‚ 123.8203125 disk.sda1.in‚0 disk.sda.out‚ 54.3515625 disk.sda3.in‚0 disk.rbd2.in‚0 disk.rbd0.out‚0 network.sent‚ 3.095444e+06) disk./.usagePercent‚36.63109020419324 disk.drbd2165.in‚0# memory_percent‚95.4566478729248* proc.memory.percent‚16.979050636291504 disk.rbd3.in‚0" disk.dm-0.out‚7254.68896484375% disk.nvme0n1.out‚7254.68896484375“Þ—Ý“– ‚(:ˆ ‚½çÒ·³à¤ gpu.0.gpu‚66.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª, — ‚(: ‚½çÒÓ­î¦ disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 124.6484375! disk.dm-0.out‚7264.7197265625& proc.memory.availableMB‚ 743.671875# proc.memory.rssMB‚ 2782.78515625 network.recv‚499506$ disk.nvme0n1.out‚7264.7197265625& disk.nvme0c0n1.out‚7264.7197265625 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚95.46098709106445 proc.cpu.threads‚17 network.sent‚ 3.15501e+06$ disk./.usageGB‚545.8276519775391 disk.drbd2165.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚36.63095170411722 disk.sda3.out‚ 55.05078125 disk.dm-1.out‚ 55.05078125 disk.rbd2.in‚0) proc.memory.percent‚16.98477268218994 disk.rbd0.in‚0 disk.sda.out‚ 55.05078125 disk.rbd1.out‚ 4.66796875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 124.6484375 cpu‚26.0298718880399 disk.rbd1.in‚0 disk.dm-0.in‚ 124.6484375 disk.rbd2.out‚0 disk.sda1.in‚0!ˆ=W‘˜ ‚(:† ‘½çÒû³æ£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02DÚ™ ‚(:ˆ ‘½çÒðÛ¥ network.sent‚ 3.157974e+06 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 125.36328125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 125.36328125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2782.78515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚7281.4052734375 disk.rbd0.in‚0 network.recv‚503107$ memory_percent‚95.46103477478027& disk.nvme0c0n1.out‚7281.4052734375 disk.sda3.out‚ 56.98046875' proc.memory.availableMB‚ 743.6640625 cpu‚26.028971410014613$ disk./.usageGB‚545.8333168029785 disk.dm-1.in‚0 disk.dm-1.out‚ 56.98046875 disk.rbd2.out‚0$ disk.nvme0n1.out‚7281.4052734375) disk./.usagePercent‚36.63133187530557 disk.rbd1.out‚ 4.76171875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 125.36328125 disk.sda.out‚ 56.98046875) proc.memory.percent‚16.98477268218994b†–2š - _wandbruntime‚1770  _runtime‚1770ÅÀ+wÈ›  trainer/global_step‚2049 $ _timestamp‚1.784274578017744e+09 _runtime‚1770.143348313 _step‚165 ' lr-AdamW/pg1‚0.00046446100428099217 ' lr-AdamW/pg2‚0.00046446100428099217fP‰`Íœ Ç _runtime‚1770.143348313 _step‚165 ' lr-AdamW/pg1‚0.00046446100428099217 ' lr-AdamW/pg2‚0.00046446100428099217 trainer/global_step‚2049 $ _timestamp‚1.784274578017744e+09¥™GpŒ2 - _wandbruntime‚1770  _runtime‚1770Ôìê’ž ÔO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚1770.144553668 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313695898906624 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70925316911616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18980942 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2167772587008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8010210 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70925211819520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313695540528128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16252790 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requestXÙï‰Med_bytes.all.allocated‚312090283004318 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚311527784980480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311527767941120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2192822553088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚311527767941120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3595731 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68732494358528ªå®ß'Ÿ ÙO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚311527767941120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3595731 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68732494358528 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚1770.144553668 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313695898906624 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70925316911616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18980942 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2167772587008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8010210 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70925211819520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313695540528128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16252790 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312090283004318 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚311527784980480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311527767941120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2192822553088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0¦ÝÑ€2  - _wandbruntime‚1770  _runtime‚1770¿/Œã&¡ ÝM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165476283702 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2168785213952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70959766421504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2169126560256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2193754734592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.784274578631205e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313844152570368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313844152570368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8014038 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4416565 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312237804730024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312237447140662 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18999985 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3597473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311675008970752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8014306 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2049 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18989994 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3597476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310071970856960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2165816833704 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18999985 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70959659239424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2169126560256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2193754734592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.784274578631205e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313844152570368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313844152570368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8014038 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4416565 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312237804730024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312237447140662 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18999985 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3597473 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311675008970752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8014306 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2049 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18989994 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3597476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310071970856960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2165816833704 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18999985 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70959659239424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165476283702 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2168785213952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70959766421504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664§©¦Ê2£ - _wandbruntime‚1770  _runtime‚1770зžwt¤  /loss/FRt2/assignment_loss‚1.237463151460099 /loss/FRt2/detection_loss‚1.4101677665701238 1loss/SRqqt1/detection_loss‚1.3141878963565652 epoch‚5 trainer/global_step‚2049 _step‚168 /loss/FBt1/detection_loss‚0.5477397454053383 /loss/FBt2/detection_loss‚0.9451942654644571 2loss/SRqqt2/assignment_loss‚0.6025322525318997 1loss/¹„’“FBt1/assignment_loss‚0.11164956475606205 /loss/FRt1/detection_loss‚0.5703161220585312 0loss/FRt1/assignment_loss‚2.1533108012376614 2loss/SRqqt1/assignment_loss‚0.6888695190032239 1loss/FBt2/assignment_loss‚0.35669282990145335 1loss/SRqqt2/detection_loss‚0.7353719597036837 &loss/total_loss‚10.673495874449099 % _timestamp‚1.7842745786340606e+09 _runtime‚1770.759466502~ù2Æ ¥ † 2loss/SRqqt2/assignment_loss‚0.6025322525318997 1loss/FBt1/assignment_loss‚0.11164956475606205 /loss/FRt1/detection_loss‚0.5703161220585312 0loss/FRt1/assignment_loss‚2.1533108012376614 2loss/SRqqt1/assignment_loss‚0.6888695190032239 1loss/FBt2/assignment_loss‚0.35669282990145335 1loss/SRqqt2/detection_loss‚0.7353719597036837 &loss/total_loss‚10.673495874449099 % _timestamp‚1.7842745786340606e+09 _runtime‚1770.759466502 /loss/FRt2/assignment_loss‚1.237463151460099 /loss/FRt2/detection_loss‚1.4101677665701238 1loss/SRqqt1/detection_loss‚1.3141878963565652 epoch‚5 trainer/global_step‚2049 _step‚168 /loss/FBt1/detection_loss‚0.5477397454053383 /loss/FBt2/detection_loss‚0.9451942654644571¨ØÁ]r¦ ‚(:’  ½çÒ¥½¤ gpu.0.gpu‚46.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LÄÎö§ ‚(:ú  ½çÒ§óæ¦ network.recv‚506274 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚96.96547985076904 disk.sda2.out‚0) proc.memory.percent‚16.98479652404785 proc.cpu.threads‚17$ disk./.usageGB‚545.8288192749023 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 126.19140625 disk.sda.in‚0 disk.sda3.out‚ 59.1015625 disk.rbd1.in‚0 disk.dm-0.out‚ 7297.6328125 network.sent‚ 3.163043e+06 disk.drbd2165.in‚0# disk.nvme0c0n1.out‚ 7297.6328125 disk.dm-0.in‚ 126.19140625" proc.memory.rssMB‚ 2782.7890625( disk./.usagePercent‚36.6310300424227 disk.rbd0.out‚0 disk.dm-1.out‚ 59.1015625 disk.rbd2.in‚0( proc.memory.availableMB‚ 497.17578125 cpu‚26.03225199481188 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 4.76171875! disk.nvme0n1.out‚ 7297.6328125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd2.out‚0 disk.sda.out‚ 59.1015625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 126.19140625‰Ýžž’¨ ‚(:‡ ¯½çÒÎöÅ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05d¹ © ‚(:‚ ¯½çÒ ¨„§ disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 126.90625 disk.rbd2.out‚0 disk.nvme0n1.in‚ 126.90625 disk.sda1.out‚0 disk.dm-1.out‚ 59.91796875 disk.rbd1.in‚0 cpu‚26.034828032499725 network.recv‚510833 disk.dm-0.in‚ 126.90625 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 4.83984375 network.sent‚ 3.228148e+06" disk.dm-0.out‚7304.17724609375' disk.nvme0c0n1.out‚7304.17724609375" proc.memory.rssMB‚ 2782.7890625) proc.memory.percent‚16.98479652404785 proc.cpu.threads‚17$ disk./.usageGB‚545.8287811279297 disk.drbd2165.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 496.98828125 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 59.91796875) disk./.usagePercent‚36.63102748234736% disk.nvme0n1.out‚7304.17724609375 disk.sda3.out‚ 59.91796875$ memory_percent‚96.96662425994873 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0™ “2ª - _wandbruntime‚1810  _runtime‚1810ãÙÛ·Æ« À _step‚169 & lr-AdamW/pg1‚0.0004639588760822489 & lr-AdamW/pg2‚0.0004639588760822489 trainer/global_step‚2099 $ _timestamp‚1.784274618541768e+09 _runtime‚1810.667385051v#X0ˬ Å _step‚169 & lr-AdamW/pg1‚0.0004639588760822489 & lr-AdamW/pg2‚0.0004639588760822489 trainer/global_step‚2099 $ _timestamp‚1.784274618541768e+09 _runtime‚1810.667385051©bØ`‰2­ - _wandbruntime‚1810  _runtime‚1810U^ Ú'® ÔO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321108287511552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16634390 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218809427968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72648063496704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318889836462080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318889819422720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16634392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19445371 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19435394 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321108287511552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19445371 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8201061 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218809427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318889819422720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70403932284928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19435394 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2215422266778 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218468088832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3682981 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70403913672704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319466075495576 trainer/global_step‚2099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321108645890048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218468088832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2244131211776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269& oÙß'¯ ÙO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2244044731904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842746185428238e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16634390 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70403932284928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19435394 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2215422266778 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218468088832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3682981 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70403913672704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319466075495576 trainer/global_step‚2099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321108645890048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218468088832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2244131211776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321108287511552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16634390 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218809427968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72648063496704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318889836462080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318889819422720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16634392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19445371 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19435394 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321108287511552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19445371 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8201061 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218809427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318889819422720 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72682511460352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚2099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321256899553792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204557 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2245150416896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19444446 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_traiñ'ó n_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2245061847040 % _timestamp‚1.7842746191567798e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317397504450560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19454437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19454437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72682511460352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚2099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321256899553792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204557 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2245150416896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19444446 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0«9Œj72³ - _wandbruntime‚1811  _runtime‚1811I®pm´ ÿ 1loss/FBt1/assignment_loss‚0.04733607266082061 2loss/FBt2/assignment_loss‚0.002094133013139299 /loss/FRt1/detection_loss‚0.5486111720298495 /loss/FRt2/detection_loss‚1.3072103605971823 epoch‚5 trainer/global_step‚2099 $ _timestamp‚1.784274619158581e+09 2loss/SRqqt1/assignment_loss‚0.6294900577044411 2loss/SRqqt2/assignment_loss‚0.6246079926280147 1loss/SRqqt1/detection_loss‚1.2640285009010273 /loss/FBt1/detection_loss‚0.7309403737291083 _runtime‚1811.283919268 _step‚172 /loss/FRt1/assignment_loss‚2.226154791489812 /loss/FRt2/assignment_loss‚1.184480149216122 1loss/SRqqt2/detection_loss‚0.7075114753980858 .loss/FBt2/detection_loss‚1.523980613425374 &loss/total_loss‚10.796445692792977Ý{mÉ µ „ &loss/total_loss‚10.796445692792977 1loss/FBt1/assignment_loss‚0.04733607266082061 2loss/FBt2/assignment_loss‚0.002094133013139299 /loss/FRt1/detection_loss‚0.5486111720298495 /loss/FRt2/detection_loss‚1.3072103605971823 epoch‚5 trainer/global_step‚2099 $ _timestamp‚1.784274619158581e+09 2loss/SRqqt1/assignment_loss‚0.6294900577044411 2loss/SRqqt2/assignment_loss‚0.6246079926280147 1loss/SRqqt1/detection_loss‚1.2640285009010273 /loss/FBt1/detection_loss‚0.7309403737291083 _runtime‚1811.283919268 _step‚172 /loss/FRt1/assignment_loss‚2.226154791489812 /loss/FRt2/assignment_loss‚1.184480149216122 1loss/SRqqt2/detection_loss‚0.7075114753980858 .loss/FBt2/detection_loss‚1.523980613425374¬=z𓶠‚(:ˆ ¾½çҺ磻 gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚253.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ì{ù· ‚(:î ¾½çÒ¯Ù­¦ disk.dm-0.in‚ 126.90625 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.availableMB‚498) disk./.usagePercent‚36.63101596200832 disk.sda2.out‚0 network.recv‚513648 disk.nvme0c0n1.in‚ 126.90625 disk.sda3.in‚0 disk.rbd1.in‚0 cpu‚26.037523082737792$ disk./.usageGB‚545.8286094665527 disk.sda3.out‚ 60.6640625 disk.nvme0n1.in‚ 126.90625* proc.memory.percent‚16.980504989624023# disk.nvme0n1.out‚7312.333984375 network.sent‚ 3.230195e+06 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.out‚7312.333984375! memory_percent‚96.96044921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 60.6640625 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 4.83984375" proc.memory.rssMB‚ 2782.0859375% disk.nvme0c0n1.out‚7312.333984375 disk.dm-1.out‚ 60.6640625}7ýs“¸ ‚(:ˆ ͽçÒ﹜¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Åxñÿ ¹ ‚(:€ ͽçÒ¥…¬§) disk./.usagePercent‚36.63113577353434 disk.rbd2.out‚0% disk.nvme0n1.out‚7319.22607421875 disk.dm-1.out‚ 60.9609375 disk.rbd1.out‚ 4.96484375 network.sent‚ 3.295366e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 127.734375$ memory_percent‚96.96462154388428( proc.memory.availableMB‚ 497.31640625 network.recv‚518040' disk.nvme0c0n1.out‚7319.22607421875 disk.sda.out‚ 60.9609375 disk.sda3.in‚0) proc.memory.percent‚16.98479652404785 disk.rbd2.in‚0 disk.nvme0n1.in‚ 127.734375 disk.drbd2165.in‚0 disk.sda3.out‚ 60.9609375 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 127.734375 disk.rbd3.out‚0" proc.memory.rssMB‚ 2782.7890625 cpu‚26.03972665936391 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.out‚0# disk./.usageGB‚545.830394744873 disk.rbd0.out‚0 disk.dm-1.in‚0" disk.dm-0.out‚7319.22607421875H¿wE“º ‚(:ˆ ܽçÒ͙Ǥ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_g¥» ‚(:ˆ ܽçÒ‘ÉÔ¦' proc.memory.availableMB‚ 498.0703125) disk./.usagePercent‚36.63115087797886! disk.dm-0.out‚7328.4833984375 disk.rbd0.in‚0 cpu‚26.042075792926077 proc.cpu.threads‚17& disk.nvme0c0n1.out‚7328.4833984375 disk.sda2.in‚0 disk.sda3.out‚ 61.12109375 disk.dm-1.in‚0 network.sent‚ 3.300436e+06 network.recv‚520904 disk.nvme0n1.in‚ 128.44921875$ disk.nvme0n1.out‚7328.4833984375 disk.sda2.out‚0 disk.rbd1.out‚ 4.96484375$ disk./.usageGB‚545.8306198120117 disk.rbd3.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2782.0859375* proc.memory.percent‚16.980504989624023 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚ 61.12109375 disk.dm-0.in‚ 128.44921875" disk.nvme0c0n1.in‚ 128.44921875 disk.rbd2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 61.12109375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.96002006530762cárú2¼ - _wandbruntime‚1851  _runtime‚1851úÞAëȽ  ' lr-AdamW/pg2‚0.00046344212796993673 trainer/global_step‚2149 $ _timestamp‚1.784274658938955e+09 _runtime‚1851.064664303 _step‚173 ' lr-AdamW/pg1‚0.00046344212796993673'o ©Í¾ Ç ' lr-AdamW/pg1‚0.00046344212796993673 ' lr-AdamW/pg2‚0.00046344212796993673 trainer/global_step‚2149 $ _timestamp‚1.784274658938955e+09 _runtime‚1851.064664303 _step‚173­–ë°2¿ - _wandbruntime‚1851  _runtime‚1851”£ûÚ'À ÔO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324576544411648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4621194 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326842583165334 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36915815 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19889846 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚328521392873472 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326251870904320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3770231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2295228423168 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚2149 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19889846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19899823 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2269504929792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265698203796 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3770228 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74370598634496 _runtime‚1851.069357482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36905836 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17015990 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328521034494976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842746589401412e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72075351599104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚174 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72075370211328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328521034494976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326842225576084 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19899823 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521392873472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17015992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2295129360384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2269163590656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚326251887943680 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8391155 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8391425 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4620927 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2269163590656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328521034494976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326842225576084 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19899823 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521392873472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17015992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2295129360384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2269163590656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚326251887943680 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8391155 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8391425 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_traiÝDž)n_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4620927 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2269163590656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19899823 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2269504929792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265698203796 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3770228 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74370598634496 _runtime‚1851.069357482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36905836 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17015990 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328521034494976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842746589401412e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72075351599104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚174®d{z³2 - _wandbruntime‚1851  _runtime‚1851§Šöã&à ÝM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324723055083520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326399111933952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326990104891040 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328669646537216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328669288151552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36932513 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8394961 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2296148343296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19898898 _runtime‚1851.678604724 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2270517563904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2270176217600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17023622 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36922520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74405040077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3771976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4623252 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74404928700928 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17023622 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8395228 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72108780357632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚326399128973312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328669646537216 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324723038044160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324723038044160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324723055083520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚326399111933952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326990104891040 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328669646537216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328669288151552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36932513 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8394961 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2296148343296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19898898 _runtime‚1851.678604724 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2270517563904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2270176217600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17023622 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36922520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74405040077312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3771976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4623252 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74404928700928 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169¯Ÿº©2Å - _wandbruntime‚1851  _runtime‚1851K ìLÆ ÿ 1loss/SRqqt1/detection_loss‚1.2518732861426227 &loss/total_loss‚10.503178002609072 epoch‚5 trainer/global_step‚2149 % _timestamp‚1.7842746595553439e+09 _runtime‚1851.680676384 /loss/FRt1/assignment_loss‚2.183534715741368 2loss/SRqqt2/assignment_loss‚0.6805081465084948 0loss/FRt2/assignment_loss‚1.2145464002339699 1loss/FBt2/assignment_loss‚0.05978233346849475 /loss/FRt1/detection_loss‚0.5702651940880157 .loss/SRqqt2/detection_loss‚0.7035527754107 /loss/FBt2/detection_loss‚1.2040364355421984 2loss/SRqqt1/assignment_loss‚0.6728590143052257 2loss/FBt1/assignment_loss‚0.036454968517331196 /loss/FRt2/detection_loss‚1.3778029334983806 /loss/FBt1/detection_loss‚0.5479617991522715 _step‚176q¤A Ç „ 2loss/FBt1/assignment_loss‚0.036454968517331196 /loss/FRt2/detection_loss‚1.3778029334983806 /loss/FBt1/detection_loss‚0.5479617991522715 _step‚176 1loss/SRqqt1/detection_loss‚1.2518732861426227 &loss/total_loss‚10.503178002609072 epoch‚5 trainer/global_step‚2149 % _timestamp‚1.7842746595553439e+09 _runtime‚1851.680676384 /loss/FRt1/assignment_loss‚2.183534715741368 2loss/SRqqt2/assignment_loss‚0.6805081465084948 0loss/FRt2/assignment_loss‚1.2145464002339699 1loss/FBt2/assignment_loss‚0.05978233346849475 /loss/FRt1/detection_loss‚0.5702651940880157 .loss/SRqqt2/detection_loss‚0.7035527754107 /loss/FBt2/detection_loss‚1.2040364355421984 2loss/SRqqt1/assignment_loss‚0.6728590143052257° &®ñ’È ‚(:‡ ë½çÒºî㦠gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àÌñùÉ ‚(:ƒ ë½çÒ¶¸Ó¬) disk./.usagePercent‚36.63119337522955 disk.dm-1.in‚0' proc.memory.availableMB‚ 498.5078125) proc.memory.percent‚16.98479652404785 network.sent‚ 3.316399e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 61.6328125 disk.rbd1.in‚0 disk.rbd2.out‚0$ disk.nvme0n1.out‚7337.3154296875 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 61.6328125 proc.cpu.threads‚17 network.recv‚524343$ disk./.usageGB‚545.8312530517578 disk.rbd3.in‚0 disk.dm-0.in‚ 129.27734375! disk.dm-0.out‚7337.3154296875 disk.nvme0n1.in‚ 129.27734375 disk.sda1.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚7337.3154296875 disk.sda.in‚0 disk.rbd1.out‚ 5.04296875 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0$ memory_percent‚96.95734977722168" proc.memory.rssMB‚ 2782.7890625 cpu‚26.04498429741866" disk.nvme0c0n1.in‚ 129.27734375 disk.sda2.out‚0 disk.dm-1.out‚ 61.6328125 disk.rbd2.in‚05}¥•“Ê ‚(:ˆ ú½çÒ°Šú£ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Ë ‚(:‰ ú½çÒ줦 disk.dm-0.in‚ 129.27734375 disk.rbd2.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚7370.63720703125 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚96.95401191711426 disk.sda3.out‚ 63.33203125 disk.rbd1.out‚ 5.04296875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 129.27734375% disk.nvme0n1.out‚7370.63720703125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚7370.63720703125 disk.drbd2165.out‚0 disk.sda.in‚0 network.sent‚ 3.368699e+06 network.recv‚528695) disk./.usagePercent‚36.63256941572609 disk.sda1.in‚0 disk.dm-1.out‚ 63.33203125' proc.memory.availableMB‚ 499.0546875 disk.rbd2.out‚0* proc.memory.percent‚16.980504989624023 disk.sda.out‚ 63.33203125" proc.memory.rssMB‚ 2782.0859375 cpu‚26.0470268192541 disk.sda2.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 129.27734375 disk.sda2.in‚0$ disk./.usageGB‚545.8517570495605 disk.rbd1.in‚0ÙDs¹žÌ ‚(:“ ‰¾çÒë€Ì£ gpu.0.gpu‚43.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n¿n2Í ‚(:‡ ‰¾çÒÙÿê¥ disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 129.9921875 disk.sda2.in‚0 disk.rbd2.out‚0 disk.dm-0.in‚ 129.9921875 proc.cpu.threads‚17 network.sent‚ 3.373776e+06% disk.nvme0n1.out‚7377.98388671875( proc.memory.availableMB‚ 498.30078125 disk.rbd0.out‚0 disk.sda.in‚0 cpu‚26.05016243859649 disk.sda.out‚ 63.93359375 disk.sda3.out‚ 63.93359375 disk.dm-1.out‚ 63.93359375 disk.rbd1.in‚0 disk.rbd1.out‚ 5.10546875) proc.memory.percent‚16.98479652404785 network.recv‚532221) disk./.usagePercent‚36.63152746506173' disk.nvme0c0n1.out‚7377.98388671875 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0$ memory_percent‚96.95861339569092" proc.memory.rssMB‚ 2782.7890625$ disk./.usageGB‚545.8362312316895 disk.nvme0n1.in‚ 129.9921875 disk.rbd3.out‚0" disk.dm-0.out‚7377.98388671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚04âÐÙ2Î - _wandbruntime‚1891  _runtime‚1891[<¹ÀÉÏ Ã % _timestamp‚1.7842746994686787e+09 _runtime‚1891.595424507 _step‚177 ' lr-AdamW/pg1‚0.00046291079313382123 ' lr-AdamW/pg2‚0.00046291079313382123 trainer/global_step‚2199ñ,\Ó7Ð È ' lr-AdamW/pg1‚0.00046291079313382123 ' lr-AJÙBq—damW/pg2‚0.00046291079313382123 trainer/global_step‚2199 % _timestamp‚1.7842746994686787e+09 _runtime‚1891.595424507 _step‚177±¬’âú2Ñ - _wandbruntime‚1891  _runtime‚1891ËrÜkÚ'Ò ÔO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2316314690704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334218733245842 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319859092480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17397590 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76093016368640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76092904985088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335934139856896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17397592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20344298 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17397592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4723773 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333613939425280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334218375656592 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20354275 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚331902078005248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚2199 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333613939425280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2346208230912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335933781478400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333613922385920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335933781478400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20344298 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3857481 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚331902060965888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20354275 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8581254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚1891.595523574 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335934139856896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3857478 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2316655240594 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17397590 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73746808137728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4723509 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8580987 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73746789525504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2346115459584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335933781478400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20344298 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3857481 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚331902060965888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20354275 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8581254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚1891.595523574 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335934139856896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3857478 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2316655240594 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17397590 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73746808137728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4723509 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8580987 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73746789525504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2346115459584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319859092480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17397590 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76093016368640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76092904985088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335934139856896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17397592 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20344298 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17397592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4723773 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333613939425280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334218375656592 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20354275 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚331902078005248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚2199 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333613939425280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2346208230912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335933781478400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333613922385920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332048571637760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73780236896256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8584771 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321213065728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20353350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4725819 _runtime‚1892.208744295 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333761180454912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334365897382186 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76127462075392 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336082393520640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321213065728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334366254971548 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320871719424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3859226 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2347130317824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336082035134976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3859223 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2320871719424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73780218284032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20363341'8Ö âM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332048571637760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73780236896256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8584771 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321213065728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20353350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4725819 _runtime‚1892.208744295 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333761180454912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334365897382186 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76127462075392 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336082393520640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321213065728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334366254971548 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320871719424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3859226 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2347130317824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336082035134976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3859223 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2320871719424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73780218284032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20363341 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17405222 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20363341 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169³4`pY2× - _wandbruntime‚1892  _runtime‚1892q–&©Ø ‚ 2loss/SRqqt2/assignment_loss‚0.6259445428019095 1loss/FBt2/assignment_loss‚0.08201933137544969 /loss/FRt2/detection_loss‚1.3532041706119473 1loss/SRqqt1/detection_loss‚1.2105837538248407 /loss/FBt2/detection_loss‚0.8228651165063011 _runtime‚1892.210765512 0loss/FRt1/assignment_loss‚2.1123245649497604 2loss/SRqqt1/assignment_loss‚0.6829201688587547 /loss/FRt1/detection_loss‚0.5480305139660134 /loss/FBt1/detection_loss‚0.6620180173573105 &loss/total_loss‚10.044975275323543 epoch‚5 0loss/FRt2/assignment_loss‚1.2015882587343398 1loss/FBt1/assignment_loss‚0.10379898149495932 1loss/SRqqt2/detection_loss‚0.6396778548419549 trainer/global_step‚2199 % _timestamp‚1.7842747000854402e+09 _step‚180­á Ù ‡ trainer/global_step‚2199 % _timestamp‚1.7842747000854402e+09 _step‚180 2loss/SRqqt2/assignment_loss‚0.6259445428019095 1loss/FBt2/assignment_loss‚0.08201933137544969 /loss/FRt2/detection_loss‚1.3532041706119473 1loss/SRqqt1/detection_loss‚1.2105837538248407 /loss/FBt2/detection_loss‚0.8228651165063011 _runtime‚1892.210765512 0loss/FRt1/assignment_loss‚2.1123245649497604 2loss/SRqqt1/assignment_loss‚0.6829201688587547 /loss/FRt1/detection_loss‚0.5480305139660134 /loss/FBt1/detection_loss‚0.6620180173573105 &loss/total_loss‚10.044975275323543 epoch‚5 0loss/FRt2/assignment_loss‚1.2015882587343398 1loss/FBt1/assignment_loss‚0.10379898149495932 1loss/SRqqt2/detection_loss‚0.6396778548419549´Õx]!žÚ ‚(:“ ˜¾çÒ‘èè¤ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*oHÑ Û ‚(: ˜¾çÒ–‘¦ disk.rbd1.in‚0$ disk.nvme0n1.out‚7394.6376953125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 5.10546875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 network.sent‚ 3.376698e+06 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2782.0859375* proc.memory.percent‚16.980504989624023 disk.dm-0.in‚ 130.8203125 disk.sda.in‚0 disk.sda.out‚ 66.0703125 cpu‚26.05233150395239 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 66.0703125! disk.dm-0.out‚7394.6376953125 disk.nvme0n1.in‚ 130.8203125& disk.nvme0c0n1.out‚7394.6376953125) disk./.usagePercent‚36.63130320246174 disk.sda2.out‚0 disk.sda3.out‚ 66.0703125 network.recv‚535644 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd2.out‚0' proc.memory.availableMB‚ 498.8203125$ disk./.usageGB‚545.8328895568848! disk.nvme0c0n1.in‚ 130.8203125$ memory_percent‚96.95544242858887 ²WОÜ ‚(:“ §¾çÒ‡«÷¤ gpu.0.gpu‚45.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.051' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.410199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?`ƒ‡Ý ‚(:ü §¾çÒÙª»§ disk.sda3.out‚66.6875* disk./.usagePercent‚36.631297826303516 disk.rbd1.out‚ 5.18359375 disk.rbd3.out‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 498.12890625" proc.memory.rssMB‚ 2782.7890625 disk.rbd2.in‚0 disk.sda2.out‚0 cpu‚26.05520266681235 proc.cpu.threads‚17! disk.dm-0.out‚7401.9111328125 disk.dm-1.out‚66.6875 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 131.53515625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 131.53515625 network.sent‚ 3.439852e+06$ disk./.usageGB‚545.8328094482422 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚0$ disk.nvme0n1.out‚7401.9111328125 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚7401.9111328125$ memory_percent‚96.95966243743896) proc.memory.percent‚16.98479652404785 network.recv‚540447 disk.nvme0n1.in‚ 131.53515625 disk.rbd0.out‚0 disk.sda.out‚66.6875 disk.sda3.in‚0OÌüZ2Þ - _wandbruntime‚1932  _runtime‚1932X3Ÿ}Çß Á trainer/global_step‚2249 % _timestamp‚1.7842747399937375e+09 _runtime‚1932.119417817 _step‚181 & lr-AdamW/pg1‚0.0004623649057005461 & lr-AdamW/pg2‚0.0004623649057005461®)†Ìà Æ % _timestamp‚1.7842747399937375e+09 _runtime‚1932.119417817 _step‚181 & lr-AdamW/pg1‚0.0004623649057005461 & lr-AdamW/pg2‚0.0004623649057005461 trainer/global_step‚2249µlÀ“2á - _wandbruntime‚1932  _runtime‚1932ý—8ÎÚ'â ÔO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2370895933440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚341594525737100 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340975973867520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4826537 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340975973867520 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343346528461824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4826806 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2370554594304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17779190 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚339227611598848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚2249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8771537 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3944728 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77815494653952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8771265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1932.120645405 _step‚182 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20798750 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2370895933440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17779192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2ß·†‡%ã ÙO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2370554594304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339227594559488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343346886840320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340975990906880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2367271727502 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17779192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340975990906880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚339227611598848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚2249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8771537 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3944728 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77815494653952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8771265 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1932.120645405 _step‚182 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20798750 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2370895933440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17779192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2370895933440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚341594525737100 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340975973867520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4826537 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340975973867520 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343346528461824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4826806 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2370554594304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17779190 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4828571 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341742405052056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75451674822656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2249 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75451656210432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842747406074817e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8775044 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20817793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2367942231334 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17786822 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8775322 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4828846 _runtime‚ 1932.73358307 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20817793 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2371567221248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343495140504064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20807802 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343495140504064 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2371908567552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17786824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339374105231360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576û>½@è&æ âM ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341123214897152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2371567221248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8775044 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20817793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2367942231334 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17786822 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8775322 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4828846 _runtime‚ 1932.73358307 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20817793 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2371567221248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343495140504064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20807802 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343495140504064 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2371908567552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17786824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339374105231360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4828571 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341742405052056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75451674822656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2249 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75451656210432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842747406074817e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350759275445248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39423971 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350759633823744 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77089683990528 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346553128153088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2421591435264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21263179 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21253202 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2448286638592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842747802688198e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2448379409920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350759633823744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77089665378304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348338042388480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4929597 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2417888214410 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚350759275445248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4031978 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18160792ŠPÀ{ß'õ ÙO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4031978 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18160792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8961295 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4031981 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350759275445248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39423971 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350759633823744 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77089683990528 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346553128153088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2421591435264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21263179 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21253202 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2448286638592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842747802688198e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2448379409920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350759633823744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77089665378304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348338042388480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4929597 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2417888214410 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚350759275445248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283º…Õàu2ö - _wandbruntime‚1973  _runtime‚1973cŽ êä&÷ ÞM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8965349 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚2299 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21272245 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2422604069376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2422262723072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4033723 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21262254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348485283418112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2422604069376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4931360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2449405559808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348485266378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18168424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21262254 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349118197543202 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 1973.00829193 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18168422 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79572518308864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2418899268244 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348485266378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8965083 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4931623 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚350907529101824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350907887487488 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2422262723072 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346699655864320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2418558718242 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348485283418112 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350907887487488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77123112749056 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39430676 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346699638824960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842747808824067e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39430676 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346699638824960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842747808824067e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21262254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348485283418112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2422604069376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4931360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2449405559808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348485266378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18168424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21262254 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349118197543202 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 1973.00829193 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18168422 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79572518308864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2418899268244 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348485266378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8965083 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4931623 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚350907529101824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350907887487488 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2422262723072 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346699655864320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2418558718242 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348485283418112 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350907887487488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77123112749056 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81260394561024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78761121916928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353878661746688 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18542390 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2472286937088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18542392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4119228 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355700093870080 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2499386124800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358172380807168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353878678786048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2468164151428 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18542390 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21717631 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9151534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40260023 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358172022428672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355700076830720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21707654 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2349ëß›ß'‡ ÙO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18542392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355700076830720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2472286937088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78761103304704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81260394561024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78761121916928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353878661746688 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18542390 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2472286937088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18542392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4119228 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355700093870080 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2499386124800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358172380807168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353878678786048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2468164151428 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18542390 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21717631 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9151534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40260023 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358172022428672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355700076830720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21707654 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2349 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496¾‚îÔd2ˆ - _wandbruntime‚2013  _runtime‚2013cnë‰ ÞM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21716706 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2472958224896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81294843817984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2469515755152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2472958224896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18550022 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18550024 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚2013.450197741 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚14‡ZÇG8612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2500400324608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358320634470912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81294951000064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355847334899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2473299571200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354025189457920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2469175205150 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18550024 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9155578 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2500311754752 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9155306 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4120973 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78794550675456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356494705213072 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚2349 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7842748213244143e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21726697 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354025172418560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358320634470912 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21716706 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18550022 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5034602 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0\£S¯é&Š ãM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2500400324608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358320634470912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81294951000064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355847334899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2473299571200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354025189457920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2469175205150 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18550024 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9155578 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2500311754752 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9155306 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4120973 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78794550675456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356494705213072 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚2349 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7842748213244143e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21726697 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354025172418560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358320634470912 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21716706 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18550022 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5034602 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21716706 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2472958224896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81294843817984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2469515755152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2472958224896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18550022 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18550024 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚2013.450197741 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272¿ö™M.2‹ - _wandbruntime‚2013  _runtime‚2013EL=½Œ € /loss/FRt1/assignment_loss‚2.105885721038986 2loss/SRqqt2/assignment_loss‚0.6266104726148596 1loss/FBt1/assignment_loss‚0.05492459744571329 2loss/FBt2/assignment_loss‚0.032278686406895456 /loss/FRt1/detection_loss‚0.5473626156510103 1loss/SRqqt1/detection_loss‚1.0276785320123283 1loss/SRqqt2/detection_loss‚0.7108052387348308 /loss/FBt1/detection_loss‚0.5405829408715573 _runtime‚2013.451740474 _step‚192 /loss/FBt2/detection_loss‚0.7956970028268794 epoch‚5 0loss/FRt2/assignment_loss‚1.2382646702778404 2loss/SRqqt1/assignment_loss‚0.6198095282946461 /loss/FRt2/detection_loss‚1.3147614721126881 %loss/total_loss‚9.614661478288234 trainer/global_step‚2349 $ _timestamp‚1.784274821326414e+09PÄŒ  … /loss/FBt1/detection_loss‚0.5405829408715573 _runtime‚2013.451740474 _step‚192 /loss/FBt2/detection_loss‚0.7956970028268794 epoch‚5 0loss/FRt2/assignment_loss‚1.2382646702778404 2loss/SRqqt1/assignment_loss‚0.6198095282946461 /loss/FRt2/detection_loss‚1.3147614721126881 %loss/total_loss‚9.614661478288234 trainer/global_step‚2349 $ _timestamp‚1.784274821326414e+09 /loss/FRt1/assignment_loss‚2.105885721038986 2loss/SRqqt2/assignment_loss‚0.6266104726148596 1loss/FBt1/assignment_loss‚0.05492459744571329 2loss/FBt2/assignment_loss‚0.032278686406895456 /loss/FRt1/detection_loss‚0.5473626156510103 1loss/SRqqt1/detection_loss‚1.0276785320123283 1loss/SRqqt2/detection_loss‚0.7108052387348308ÀúŠ0žŽ ‚(:“ ¿çÒƒ†ã¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êŒ,Ê  ‚(:‚ ¿çÒ”ÅΧ disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 cpu‚26.06845735672091 proc.cpu.threads‚17 disk.rbd2.out‚0 disk.nvme0n1.in‚ 135.44921875 network.recv‚566408 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 5.44921875" proc.memory.rssMB‚ 2782.7890625& disk.nvme0c0n1.out‚7459.5908203125 disk.sda2.in‚0 disk.sda3.out‚ 69.1640625 disk.dm-1.out‚ 69.1640625! disk.dm-0.out‚7459.5908203125 disk.rbd0.out‚0$ memory_percent‚96.96130752563477 disk.dm-0.in‚ 135.44921875$ disk.nvme0n1.out‚7459.5908203125 disk.sda.out‚ 69.1640625 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 network.sent‚ 3.604414e+06) disk./.usagePercent‚36.63237741007541 disk.sda.in‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 497.859375$ disk./.usageGB‚545.8488960266113 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 135.44921875 disk.rbd3.in‚0) proc.memory.percent‚16.98479652404785™Oi” ‚(:’ Ÿ¿çÒØ ¡¤ gpu.0.gpu‚90.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø%Xk‘ ‚(:ƒ Ÿ¿çÒߪզ network.recv‚570818 disk.drbd2165.in‚0 disk.sda3.out‚ 69.44140625 disk.rbd3.out‚0 disk.dm-0.in‚ 136.1640625) proc.memory.percent‚16.98479652404785 network.sent‚ 3.65441e+06$ disk.nvme0n1.out‚7467.3818359375 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd2.out‚0! disk.nvme0c0n1.in‚ 136.1640625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 497.9140625 disk.sda1.out‚0 disk.sda.out‚ 69.44140625& disk.nvme0c0n1.out‚7467.3818359375) disk./.usagePercent‚36.63115907021996 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚96.96097373962402$ disk./.usageGB‚545.8307418823242 disk.dm-1.in‚0 disk.rbd1.out‚ 5.52734375" proc.memory.rssMB‚ 2782.7890625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 136.1640625 disk.dm-1.out‚ 69.44140625 disk.rbd1.in‚0! disk.dm-0.out‚7467.3818359375 cpu‚26.0706277808105479ÆØî2’ - _wandbruntime‚2053  _runtime‚2053û!5Æ“ À % _timestamp‚1.7842748611797707e+09 _runtime‚ 2053.30531803 _step‚193 & lr-AdamW/pg1‚0.0004606402830909194 & lr-AdamW/pg2‚0.0004606402830909194 trainer/global_step‚2399ÙŽË” Å & lr-AdamW/pg2‚0.0004606402830909194 trainer/global_step‚2399 % _timestamp‚1.7842748611797707e+09 _runtime‚ 2053.30531803 _step‚193 & lr-AdamW/pg1‚0.0004606402830909194Á²ô2• - _wandbruntime‚2053  _runtime‚20535d\þÚ'– ÔO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2522982438912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2522641099776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82982884588032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365585127790592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18923990 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363062128312320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4206478 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365584769412096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126063616 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82982758524416 $ _timestamp‚1.784274861180809e+09 _step‚194 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18923992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2550217293312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2522641099776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22172083 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365585127790592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 I4Nà"BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365584769412096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126063616 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82982758524416 $ _timestamp‚1.784274861180809e+09 _step‚194 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18923992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2550217293312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2522641099776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22172083 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚365585127790592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2523995073024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41102780 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2523653726720 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9345136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚195 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363209369341952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18931624 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22181149 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365733381454336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363209386381312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83017205314560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2523653726720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80465969989632 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 $ _timestamp‚1.784274861794727e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4208226 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551235324928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2523995073024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361350723051520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚365733381454336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18931624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚2399 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41102780 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2520132242060 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363209386381312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5136913 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5137196 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363209369341952 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9345422 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363870497704218 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18931622 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22181149 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22171158 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22171158 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9345136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚195 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363209369341952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18931624 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22181149 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365733381454336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363209386381312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83017205314560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2523653726720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80465969989632 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 $ _timestamp‚1.784274861794727e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4208226 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551235324928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2523995073024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361350723051520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚365733381454336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18931624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚2399 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41102780 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2520132242060 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363209386381312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5136913 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5137196 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363209369341952 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9345422 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363870497704218 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18931622 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22181149 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22171158 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22171158 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2523995073024Ã4„gÀ2› - _wandbruntime‚2053  _runtime‚2053Ç1œ ÿ 0loss/FRt2/assignment_loss‚1.2155933841180302 1loss/SRqqt1/detection_loss‚1.1034429396355623 trainer/global_step‚2399 _step‚196 2loss/SRqqt1/assignment_loss‚0.7023442964176492 1loss/FBt1/assignment_loss‚0.07444148179430732 /loss/FRt1/detection_loss‚0.5279263741355417 0loss/SRqqt2/detection_loss‚0.667326354665916 epoch‚5 _runtime‚2053.922102855 /loss/FRt1/assignment_loss‚2.089558192856829 1loss/FBt2/assignment_loss‚0.20477402172867154 /loss/FRt2/detection_loss‚1.2903337013187335 /loss/FBt1/detection_loss‚0.4432807857853671 /loss/FBt2/detection_loss‚0.7450249404818924 2loss/SRqqt2/assignment_loss‚0.6753157970054917 %loss/total_loss‚9.739362269943992 % _timestamp‚1.7842748617967236e+09£FY»  „ 2loss/SRqqt2/assignment_loss‚0.6753157970054917 %loss/total_loss‚9.739362269943992 % _timestamp‚1.7842748617967236e+09 0loss/FRt2/assignment_loss‚1.2155933841180302 1loss/SRqqt1/detection_loss‚1.1034429396355623 trainer/global_step‚2399 _step‚196 2loss/SRqqt1/assignment_loss‚0.7023442964176492 1loss/FBt1/assignment_loss‚0.07444148179430732 /loss/FRt1/detection_loss‚0.5279263741355417 0loss/SRqqt2/detection_loss‚0.667326354665916 epoch‚5 _runtime‚2053.922102855 /loss/FRt1/assignment_loss‚2.089558192856829 1loss/FBt2/assignment_loss‚0.20477402172867154 /loss/FRt2/detection_loss‚1.2903337013187335 /loss/FBt1/detection_loss‚0.4432807857853671 /loss/FBt2/detection_loss‚0.7450249404818924Ä›:º÷‘ž ‚(:† ®¿çÒÍ줥 gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ȹ —Ÿ ‚(:ƒ ®¿çÒ‘ ¯§ disk.sda3.out‚ 69.74609375 disk.dm-0.in‚ 136.9921875 disk.sda1.out‚0$ memory_percent‚96.96393013000488' proc.memory.availableMB‚ 497.4296875 proc.memory.rssMB‚ 2783.40625! disk.nvme0c0n1.in‚ 136.9921875 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 136.9921875$ disk.nvme0n1.out‚7476.7138671875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 3.672534e+06 network.recv‚574155 disk.sda.out‚ 69.74609375 disk.dm-1.in‚0 disk.rbd1.out‚ 5.52734375! disk.dm-0.out‚7476.7138671875 disk.rbd0.out‚0* proc.memory.percent‚16.988563537597656$ disk./.usageGB‚545.8327598571777 disk.dm-1.out‚ 69.74609375 cpu‚26.073793901545937& disk.nvme0c0n1.out‚7476.7138671875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚36.63129449820557 disk.sda.in‚0 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0Çõææ  ‚(:’ ½¿çÒáÙ–¤ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mqˆY¡ ‚(:‚ ½¿çÒЂ›© disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 496.7421875$ disk./.usageGB‚545.8332061767578! disk.dm-0.out‚7483.1728515625 disk.nvme0n1.in‚ 137.70703125 disk.dm-1.out‚ 69.890625 proc.cpu.threads‚17 disk.sda1.out‚0 cpu‚26.100048627284355* Šf´disk./.usagePercent‚36.631324451087075 disk.rbd0.in‚0 disk.sda.out‚ 69.890625 disk.dm-1.in‚0 network.sent‚ 3.722492e+06 disk.rbd0.out‚0 disk.rbd1.in‚0 network.recv‚578875 disk.rbd2.in‚0$ disk.nvme0n1.out‚7483.1728515625 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚96.96812629699707" disk.nvme0c0n1.in‚ 137.70703125& disk.nvme0c0n1.out‚7483.1728515625 disk.sda3.in‚0 disk.sda3.out‚ 69.890625 disk.rbd1.out‚ 5.58984375 disk.rbd3.out‚0! proc.memory.rssMB‚ 2784.109375* proc.memory.percent‚16.992855072021484 disk.dm-0.in‚ 137.70703125 disk.sda2.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 Ð[n2¢ - _wandbruntime‚2071  _runtime‚2071±Á=£ »{ 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6825961470603943 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.676003634929657 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6166306734085083 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.41989225149154663 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47836294770240784 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5651828646659851 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6309492588043213 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6328926086425781 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6297371983528137 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +particle/specificity‚0.9066815376281738 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4111127257347107 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7486416697502136 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.048903681337833405 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5807291269302368 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6267753839492798 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03262445703148842 *jet/accuracy_1_of_3‚0.9251317381858826 *jet/accuracy_2_of_4‚0.7644436955451965 .particle/accuracy_3_of_4‚0.997619092464447 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959363341331482 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6905140280723572 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941307902336121 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6823449730873108 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30479681491851807 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6698829531669617 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_2_of_3‚0.5326505899429321 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38719677925109863 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7495936155319214 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5902863144874573 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5166666507720947 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48139235377311707 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08437999337911606 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8497570753097534 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08367732167243958 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5555353760719299 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37637439370155334 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18861077725887299 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7155763506889343 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08975263684988022 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 _step‚197 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.234035924077034 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6844207048416138 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9945365190505981 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7871018648147583 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8041571378707886 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24994641542434692 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6405729055404663 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0885176882147789 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9919062852859497 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04918654263019562 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6532319188117981 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9964284300804138 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7881091833114624 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11499999463558197 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.797713577747345 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17098820209503174 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9875646233558655 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2907264530658722 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20801445841789246 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6120145320892334 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6328926086425781 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2447916716337204 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22350028157234192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9973896741867065 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13714554905891418 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6068817377090454 +jet/accuracy_1_of_1‚0.33251500129699707 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6282013654708862 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚2399 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7515953779220581 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2223728895187378 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21143262088298798 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15360814332962036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.009931244887411594 +particle/sensitivity‚0.6142300367355347 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8088246583938599 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7537792921066284 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8129062652587891 *jet/accuracy_1_of_4‚0.9619364142417908 /particle/accuracy_3_of_3‚0.9994933605194092 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9929636716842651 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24830086529254913 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.518608808517456 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7832704186439514 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2918098270893097 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6797561645507812 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971503019332886 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2588735818862915 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7061503529548645 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2334863543510437 /particle/accuracy_4_of_4‚0.9565828442573547 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6183531880378723 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038635410368442535 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23977410793304443 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3726828694343567 $ _timestamp‚1.784274879754643e+09 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19677703082561493 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 )loss/val_total_loss‚9.555506328016431 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2472561150789261 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964209794998169 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.670353353023529 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6306357383728027 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24825425446033478 +jet/accuracy_3_of_3‚0.06738931685686111 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1379021853208542 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22363653779029846 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2595313787460327 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06119871139526367 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07500801980495453 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9952632188796997 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37587180733680725 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_1_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7391526103019714 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2118639051914215 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24830086529254913 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07500801980495453 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7907785773277283 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12166322022676468 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9180833697319031 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916428923606873 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.797713577747345 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7958003282546997 particle/accuracy_1_of_4‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16788837313652039 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6248787641525269 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _runtime‚2071.880992109 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25680842995643616 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7857489585876465 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8706108927726746 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9935552477836609 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7375965118408203 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3257805109024048 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8580056428909302 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9964284300804138 +jet/accuracy_3_of_4‚0.06799862533807755 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6200191974639893 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3748875558376312 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7012418508529663 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02630574442446232 *jet/accuracy_2_of_2‚0.1374056041240692 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8438739776611328 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9866601824760437 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚0 epoch‚5 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5121110081672668 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7223137617111206 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0¾…ùÆ=¤ À{ ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7515953779220581 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2223728895187378 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21143262088298798 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15360814332962036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.009931244887411594 +particle/sensitivity‚0.6142300367355347 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8088246583938599 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7537792921066284 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8129062652587891 *jet/accuracy_1_of_4‚0.9619364142417908 /particle/accuracy_3_of_3‚0.9994933605194092 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9929636716842651 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24830086529254913 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.518608808517456 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7832704186439514 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2918098270893097 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6797561645507812 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971503019332886 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2588735818862915 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7061503529548645 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2334863543510437 /particle/accuracy_4_of_4‚0.9565828442573547 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6183531880378723 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038635410368442535 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23977410793304443 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3726828694343567 $ _timestamp‚1.784274879754643e+09 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19677703082561493 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 )loss/val_total_loss‚9.555506328016431 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2472561150789261 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964209794998169 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.670353353023529 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6306357383728027 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24825425446033478 +jet/accuracy_3_of_3‚0.06738931685686111 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1379021853208542 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22363653779029846 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2595313787460327 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06119871139526367 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07500801980495453 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9952632188796997 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37587180733680725 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_1_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7391526103019714 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2118639051914215 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24830086529254913 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07500801980495453 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7907785773277283 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12166322022676468 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9180833697319031 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916428923606873 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.797713577747345 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7958003282546997 particle/accuracy_1_of_4‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16788837313652039 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6248787641525269 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _runtime‚2071.880992109 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25680842995643616 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7857489585876465 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8706108927726746 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9935552477836609 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7375965118408203 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3257805109024048 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8580056428909302 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9964284300804138 +jet/accuracy_3_of_4‚0.06799862533807755 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6200191974639893 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3748875558376312 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7012418508529663 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02630574442446232 *jet/accuracy_2_of_2‚0.1374056041240692 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8438739776611328 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9866601824760437 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚0 epoch‚5 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5121110081672668 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7223137617111206 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6825961470603943 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.676003634929657 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6166306734085083 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.41989225149154663 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47836294770240784 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5651828646659851 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6309492588043213 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6328926086425781 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6297371983528137 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +particle/specificity‚0.9066815376281738 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4111127257347107 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7486416697502136 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.048903681337833405 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5807291269302368 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6267753839492798 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03262445703148842 *jet/accuracy_1_of_3‚0.9251317381858826 *jet/accuracy_2_of_4‚0.7644436955451965 .particle/accuracy_3_of_4‚0.997619092464447 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9959363341331482 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6905140280723572 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941307902336121 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6823449730873108 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30479681491851807 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6698829531669617 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_2_of_3‚0.5326505899429321 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38719677925109863 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7495936155319214 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5902863144874573 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5166666507720947 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48139235377311707 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08437999337911606 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8497570753097534 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08367732167243958 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5555353760719299 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37637439370155334 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18861077725887299 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7155763506889343 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08975263684988022 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 _step‚197 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.234035924077034 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6844207048416138 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9945365190505981 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7871018648147583 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8041571378707886 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24994641542434692 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6405729055404663 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0885176882147789 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9919062852859497 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04918654263019562 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6532319188117981 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9964284300804138 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7881091833114624 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11499999463558197 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.797713577747345 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17098820209503174 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9875646233558655 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2907264530658722 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20801445841789246 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6120145320892334 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6328926086425781 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2447916716337204 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22350028157234192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9973896741867065 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13714554905891418 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6068817377090454 +jet/accuracy_1_of_1‚0.33251500129699707 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6282013654708862 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚2399ÅÒØÐp¥ ‚(:ˆ Ì¿çÒžò¯¥ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0#ÎX2# gpu.0.uncorrectedMemoryErrors‚0” é¦ ‚(:‹ Ì¿çÒñ뀧$ memory_percent‚96.97983264923096( proc.memory.availableMB‚ 494.82421875 disk.rbd2.out‚0) disk./.usagePercent‚36.63124508875146 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 137.70703125 disk.rbd1.out‚ 5.58984375 cpu‚26.102569304224033 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 70.08203125* proc.memory.percent‚16.992855072021484 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2784.109375 network.sent‚ 3.740602e+06 disk.drbd2165.out‚0 disk.dm-1.out‚ 70.08203125 disk.nvme0n1.in‚ 137.70703125$ disk./.usageGB‚545.8320236206055 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚7734.03955078125 disk.sda2.out‚0 disk.sda3.out‚ 70.08203125 disk.rbd3.in‚0" disk.dm-0.out‚7734.03955078125 network.recv‚581983 disk.rbd0.out‚0 disk.dm-0.in‚ 137.70703125 proc.cpu.threads‚17% disk.nvme0n1.out‚7734.03955078125ýj “§ ‚(:ˆ Û¿çÒ–óÖ¦ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bå±x¨ ‚(:† Û¿çÒ Ñ¡¨ disk.rbd0.out‚0 disk.sda.out‚ 70.4609375 disk.rbd3.in‚0" disk.dm-0.out‚7987.36572265625& proc.memory.availableMB‚ 495.390625 disk.rbd1.out‚ 5.66796875% disk.nvme0n1.out‚7987.36572265625 disk.rbd0.in‚0 disk.dm-1.out‚ 70.4609375 cpu‚26.103916828951164 network.sent‚ 3.787503e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 138.53515625 disk.nvme0n1.in‚ 138.53515625 disk.sda1.in‚0 network.recv‚586597 disk.sda.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.988849639892578) disk./.usagePercent‚36.63134595571995$ disk./.usageGB‚545.8335266113281 disk.sda3.out‚ 70.4609375 disk.rbd2.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2783.453125" disk.nvme0c0n1.in‚ 138.53515625 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.97637557983398' disk.nvme0c0n1.out‚7987.36572265625 proc.cpu.threads‚17Ýqh2© - _wandbruntime‚2113  _runtime‚2113N¶Æòɪ à _runtime‚2113.692335549 _step‚198 ' lr-AdamW/pg1‚0.00046003654519502653 ' lr-AdamW/pg2‚0.00046003654519502653 trainer/global_step‚2449 % _timestamp‚1.7842749215667388e+09ë¡Î« È ' lr-AdamW/pg2‚0.00046003654519502653 trainer/global_step‚2449 % _timestamp‚1.7842749215667388e+09 _runtime‚2113.692335549 _step‚198 ' lr-AdamW/pg1‚0.00046003654519502653ÆAèÃÈ2¬ - _wandbruntime‚2113  _runtime‚2113¶Ä÷NÚ'­ ÔO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374945174632448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5272154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373025508998446 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2591411148800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22696862 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2591069809664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5271893 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372353746444288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2591069809664 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84739432527872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2619434081280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2619522658304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 % _timestamp‚1.7842749215677853e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374944816253952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9568405 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2587116850478 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42113141 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19426258 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22696862 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82119909869568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82119891257344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4296251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374945174632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372353746444288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374944816253952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372353763483648 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84739325338624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9568141 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚370438409187328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2449 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4296248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2113.693585559 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19426256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372353763483648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373025866587696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2587457400368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22686885 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370438392147968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2591411148800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374944816253952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9568405 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2587116850478 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42113141 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19426258 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22696862 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82119909869568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82119891257344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4296251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374945174632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372353746444288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374944816253952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372353763483648 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84739325338624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9568141 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚370438409187328 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2449 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4296248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2113.693585559 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19426256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372353763483648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373025866587696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2587457400368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22686885 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370438392147968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2591411148800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5272154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373025508998446 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2591411148800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22696862 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2591069809664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5271893 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372353746444288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2591069809664 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84739432527872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2619434081280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2619522658304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Ǭzý2¯ - _wandbruntime‚2114  _runtime‚2114ðo¬ñé#° ÝM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82153338628096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2620451761664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433890 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2449 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22695937 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372501004513280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2588468454202 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5273872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84773771777536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚375093069910528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375093428296192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2592423782912 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9571865 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372500987473920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375093428296192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370584919859200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2592082436608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 2114.30649598 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82153320015872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372500987473920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5274137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373173030724040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842749221805754e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372501004513280 REDeviceStaí,ÓXútsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19433890 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372500987473920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5274137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373173030724040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842749221805754e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372501004513280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19433890 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2620451761664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433890 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚2449 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22695937 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372501004513280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2588468454202 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5273872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84773771777536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚375093069910528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375093428296192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2592423782912 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9571865 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372500987473920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375093428296192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370584919859200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2592082436608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42139818 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 2114.30649598 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82153320015872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23141337 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚203 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2642106650624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5376233 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382357563237376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23151314 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86462033065472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23151314 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4383498 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377763942780928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚2499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86462131866112 _runtime‚2153.955614338 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23141337 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 $ _timestamp‚1.784274961829803e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379715814965248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379715797925888 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382357563237376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382357921615872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2641765311488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19807856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380401659078954 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9759734 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379715814965248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19807856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2642106650624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379715797925888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2638073887276 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136ýZŽÀ ×O dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83791347795968 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382357921615872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2641765311488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2670784070144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9759463 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚377763925741568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382357563237376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382357921615872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2641765311488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19807856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380401659078954 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9759734 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379715814965248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19807856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2642106650624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379715797925888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2638073887276 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.smalýh8Œ l_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2637733337386 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23141337 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚203 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2642106650624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5376233 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382357563237376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23151314 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86462033065472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23151314 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4383498 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚377763942780928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚2499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86462131866112 _runtime‚2153.955614338 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23141337 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 $ _timestamp‚1.784274961829803e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379715814965248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379715797925888 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚377910453452800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23150389 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2643119284736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379863038955520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19815490 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚377910436413440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382505816893952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86496585613312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4385246 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2642777938432 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5378328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9763574 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83824757942272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382506175279616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2671809058816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2671722586112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380549538393910 trainer/global_step‚2499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23160380 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83824776554496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379863055994880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2642777938432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842749626317623e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19815488 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23160380 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9763315 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5378072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888D (¦é&à ãM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2642777938432 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5378328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9763574 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83824757942272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382506175279616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2671809058816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2671722586112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380549538393910 trainer/global_step‚2499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23160380 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83824776554496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379863055994880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2642777938432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842749626317623e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19815488 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23160380 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9763315 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5378072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19815490 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2643119284736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2639084941110 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379863038955520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380549180804548 _runtime‚2154.757627696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚377910453452800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23150389 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2643119284736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379863038955520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19815490 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚377910436413440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382505816893952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86496585613312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4385246 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ì\²ó…2Ä - _wandbruntime‚2154  _runtime‚2154‹×[ªÅ ÿ 1loss/SRqqt2/detection_loss‚0.7113816747963284 % _timestamp‚1.7842749626345267e+09 /loss/FRt1/assignment_loss‚1.978675516231298 0loss/FRt2/assignment_loss‚1.1994379380308025 1loss/FBt2/assignment_loss‚0.06374716890495151 _step‚205 2loss/SRqqt1/assignment_loss‚0.6545659511188127 2loss/SRqqt2/assignment_loss‚0.6257357715092736 1loss/FBt1/assignment_loss‚0.08150302202282467 /loss/FRt1/detection_loss‚0.5784935148371152 1loss/SRqqt1/detection_loss‚1.0981625216081738 .loss/FBt1/detection_loss‚0.477162680640218 _runtime‚2154.759886631 /loss/FRt2/detection_loss‚1.3461217059925021 /loss/FBt2/detection_loss‚0.9467856631510787 %loss/total_loss‚9.761773128843378 epoch‚6 trainer/global_step‚2499Ѐ<ä Æ „ 2loss/SRqqt1/assignment_loss‚0.6545659511188127 2loss/SRqqt2/assignment_loss‚0.6257357715092736 1loss/FBt1/assignment_loss‚0.08150302202282467 /loss/FRt1/detection_loss‚0.5784935148371152 1loss/SRqqt1/detection_loss‚1.0981625216081738 .loss/FBt1/detection_loss‚0.477162680640218 _runtime‚2154.759886631 /loss/FRt2/detection_loss‚1.3461217059925021 /loss/FBt2/detection_loss‚0.9467856631510787 %loss/total_loss‚9.761773128843378 epoch‚6 trainer/global_step‚2499 1loss/SRqqt2/detection_loss‚0.7113816747963284 % _timestamp‚1.7842749626345267e+09 /loss/FRt1/assignment_loss‚1.978675516231298 0loss/FRt2/assignment_loss‚1.1994379380308025 1loss/FBt2/assignment_loss‚0.06374716890495151 _step‚205ͯHYž‘Ç ‚(:† —ÀçÒâþÛ¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.279' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éºŽÈ ‚(:û —ÀçÒäí—§ disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0& proc.memory.availableMB‚ 495.234375! proc.memory.rssMB‚ 2783.453125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 network.sent‚ 3.862693e+06 disk.sda1.out‚0* proc.memory.percent‚16.988849639892578 network.recv‚600008 disk.nvme0n1.in‚ 140.79296875 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.63232390450075$ disk./.usageGB‚545.8480987548828! disk.nvme0n1.out‚ 8020.7578125" disk.nvme0c0n1.in‚ 140.79296875 disk.dm-1.in‚0$ memory_percent‚96.97732925415039 cpu‚26.1110919069446 disk.sda3.in‚0 disk.sda3.out‚ 71.41015625 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 8020.7578125 disk.sda.out‚ 71.41015625 disk.rbd1.in‚0 disk.dm-0.in‚ 140.79296875 disk.dm-0.out‚ 8020.7578125 disk.sda2.in‚0 disk.dm-1.out‚ 71.41015625 disk.rbd1.out‚ 5.87109375 disk.rbd2.out‚0ži–“É ‚(:ˆ ¦ÀçÒâ÷‹¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ȶ~Ê ‚(:„ ¦ÀçÒ龨§ disk.sda1.out‚0" disk.nvme0c0n1.in‚ 141.62109375 disk.rbd1.out‚ 5.87109375 disk.dm-1.out‚ 71.6328125 network.recv‚604142 disk.dm-0.in‚ 141.62109375 disk.nvme0n1.in‚ 141.62109375 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 71.6328125 disk.dm-1.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 494.47265625* proc.memory.percent‚16.992855072021484 network.sent‚ 3.925614e+06 disk.rbd2.out‚0$ memory_percent‚96.98197841644287! proc.memory.rssMB‚ 2784.109375 cpu‚26.11225193905301 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 71.6328125 disk.sda3.in‚0$ disk./.usageGB‚545.8491439819336& disk.nvme0c0n1.out‚8032.0166015625) disk./.usagePercent‚36.63239405056513 disk.rbd3.out‚0 disk.sda1.in‚0! disk.dm-0.out‚8032.0166015625$ disk.nvme0n1.out‚8032.0166015625ÊD­›Ë ‚(: µÀçÒ³¡õ£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚283.318' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.66359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¤Ì ‚(:„ µÀçÒÈËÊ¥! disk.nvme0c0n1.in‚ 142.3359375 disk.sda.out‚ 71.86328125 disk.sda2.out‚0' proc.memory.availableMB‚ 495.2734375 network.recv‚607683 disk.drbd2165.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚8040.0361328125 disk.rbd2.in‚0 disk.rbd2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.63252896653567 disk.rbd0.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚8040.0361328125 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 5.93359375 disk.rbd3.in‚0 disk.dm-0.in‚ 142.3359375 disk.sda1.in‚0 disk.sda3.out‚ 71.86328125 disk.dm-1.out‚ 71.86328125 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0! disk.dm-0.out‚8040.0361328125 network.sent‚ 3.932781e+06 disk.dm-1.in‚0$ memory_percent‚96.97709083557129! proc.memory.rssMB‚ 2783.453125$ disk./.usageGB‚545.8511543273926 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 142.3359375* proc.memory.percent‚16.988849639892578 cpu‚26.114001411972048²ò02Í - _wandbruntime‚2194  _runtime‚2194˜P£ÐÃÎ ½ _runtime‚2194.471347447 _step‚206 $ lr-AdamW/pg1‚0.00045878600513427 $ lr-AdamW/pg2‚0.00045878600513427 trainer/global_step‚2549 % _timestamp‚1.7842750023457274e+09ï¸ÌùÈÏ Â $ lr-AdamW/pg1‚0.00045878600513427 $ lr-AdamW/pg2‚0.00045878600513427 trainer/global_step‚2549 % _timestamp‚1.7842750023457274e+09 _runtime‚2194.471347447 _step‚206ÎošÖ2Ð - _wandbruntime‚2194  _runtime‚2194Fåð¦PÑ ÔO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2688349824294 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88185002635776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389770668599296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387077866446848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480229 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23605766 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842750023467891e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23605766 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2692460813312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387777809159462 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20189456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2692802152448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692460813312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389770310220800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88184899640832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387778166748712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4470748 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2722216913408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9951238 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387077866446848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23595789 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚387077849407488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23595789 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85462767110144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20189458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20189456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85462785722368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2688690374184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 kÖñß'Ò ÙO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2688349824294 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88185002635776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389770668599296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387077866446848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480229 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23605766 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842750023467891e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23605766 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2692460813312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387777809159462 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20189456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2692802152448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692460813312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389770310220800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88184899640832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387778166748712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4470748 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2722216913408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9951238 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387077866446848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23595789 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚387077849407488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23595789 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85462767110144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20189458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20189456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85462785722368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2688690374184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360ÏËæøŽ2Ó - _wandbruntime‚2195  _runtime‚2195'ïtä&Ô ÞM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387225107476480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85496195868672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389918563877376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2693814786560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20197088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2689360878016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2693473440256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85496214480896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389918922263040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88219454654464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2693814786560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9955068 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387925688474418 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387925330885056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385235987046400 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88219351666688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2195.086516265 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23614832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23604841 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4472496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387225107476480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9954805 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2689701428018 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20197088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2549 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2693473440256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85496214480896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389918922263040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88219454654464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2693814786560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9955068 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387925688474418 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_‹W&‹batch_end/allocation.all.freed‚43801929 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387925330885056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385235987046400 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88219351666688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2195.086516265 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23614832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23604841 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4472496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387225107476480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9954805 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2689701428018 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20197088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2549 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÐWSQ2Ö - _wandbruntime‚2195  _runtime‚2195“öâ× ÿ 2loss/SRqqt1/assignment_loss‚0.5906946064486099 1loss/FBt1/assignment_loss‚0.05610833276449662 /loss/FRt1/detection_loss‚0.5400421304226192 1loss/SRqqt2/detection_loss‚0.7025921456427372 % _timestamp‚1.7842750029627964e+09 _runtime‚2195.088143677 /loss/FRt1/assignment_loss‚2.025883621940043 /loss/FRt2/assignment_loss‚1.112871158577353 2loss/SRqqt2/assignment_loss‚0.6384034606345647 %loss/total_loss‚9.539470350388797 trainer/global_step‚2549 /loss/FBt2/detection_loss‚0.9994946892838925 2loss/FBt2/assignment_loss‚0.004588245070493713 /loss/FRt2/detection_loss‚1.3895531032329929 0loss/SRqqt1/detection_loss‚1.026891592476103 /loss/FBt1/detection_loss‚0.4523472638948905 _step‚209 epoch‚6÷H©) Ø „ /loss/FBt2/detection_loss‚0.9994946892838925 2loss/FBt2/assignment_loss‚0.004588245070493713 /loss/FRt2/detection_loss‚1.3895531032329929 0loss/SRqqt1/detection_loss‚1.026891592476103 /loss/FBt1/detection_loss‚0.4523472638948905 _step‚209 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5906946064486099 1loss/FBt1/assignment_loss‚0.05610833276449662 /loss/FRt1/detection_loss‚0.5400421304226192 1loss/SRqqt2/detection_loss‚0.7025921456427372 % _timestamp‚1.7842750029627964e+09 _runtime‚2195.088143677 /loss/FRt1/assignment_loss‚2.025883621940043 /loss/FRt2/assignment_loss‚1.112871158577353 2loss/SRqqt2/assignment_loss‚0.6384034606345647 %loss/total_loss‚9.539470350388797 trainer/global_step‚2549ÑÿLÞžÙ ‚(:“ ÄÀçÒ°ÏÕ¥ gpu.0.gpu‚94.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶õðÚ ‚(:û ÄÀçÒÇËú§ disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 143.1640625$ memory_percent‚96.98319435119629 disk.sda3.in‚0 disk.sda3.out‚ 72.1953125 disk.dm-0.in‚ 143.1640625 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 72.1953125$ disk./.usageGB‚545.8491554260254 disk.sda2.in‚0' proc.memory.availableMB‚ 494.2734375! proc.memory.rssMB‚ 2784.109375 cpu‚26.115183359087805 disk.nvme0n1.in‚ 143.1640625$ disk.nvme0c0n1.out‚ 8050.94140625 network.recv‚611205) disk./.usagePercent‚36.63239481858774 disk.dm-0.out‚ 8050.94140625 network.sent‚ 3.948739e+06 disk.rbd1.in‚0 disk.rbd3.out‚0" disk.nvme0n1.out‚ 8050.94140625 disk.drbd2165.out‚0* proc.memory.percent‚16.992855072021484 disk.dm-1.out‚ 72.1953125 disk.rbd1.out‚ 5.93359375 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0¢¹7‘Û ‚(:† ÓÀçұ룤 gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€‚ Ü ‚(: ÓÀçÒ±ÒƦ disk.rbd2.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 495.0234375 network.recv‚615817 disk.rbd0.in‚0 disk.sda2.in‚0! disk.dm-0.out‚8054.2275390625) disk./.usagePercent‚36.63239481858774 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚8054.2275390625 disk.dm-1.in‚0 cpu‚26.116992983316578$ disk./.usageGB‚545.8491554260254 disk.sda3.out‚ 72.3671875 disk.rbd1.out‚ 5.99609375 disk.rbd3.in‚0 network.sent‚ 3.998654e+06 disk.nvme0n1.in‚ 143.1640625 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚96.97861671447754! proc.memory.rssMB‚ 2783.453125 disk.rbd2.in‚0$ disk.nvme0n1.out‚8054.2275390625! disk.nvme0c0n1.in‚ 143.1640625 disk.sda.in‚0 disk.sda.out‚ 72.3671875 disk.sda2.out‚0 disk.dm-1.out‚ 72.3671875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 143.1640625* proc.memory.percent‚16.988849639892578 proc.cpu.threads‚17&ÖN‡‘Ý ‚(:† âÀçÒ‡‹Ò¤ gpu.0.gpu‚89.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¡5âÞ ‚(:ø âÀçÒˆèüª disk.rbd1.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 494.26171875 network.sent‚ 4.005829e+06 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚143.6875$ memory_percent‚96.98326587677002 cpu‚26.11885739083218 disk.rbd3.out‚0! disk.dm-0.out‚8063.5380859375 disk.rbd2.in‚0! proc.memory.rssMB‚ 2784.109375* proc.memory.percent‚16.992855072021484 disk.sda3.out‚ 72.5390625 disk.dm-0.in‚143.6875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 72.5390625 disk.rbd2.out‚0 disk.nvme0n1.in‚143.6875 disk.rbd0.in‚0$ disk./.usageGB‚545.8492698669434 disk.sda2.out‚0$ disk.nvme0n1.out‚8063.5380859375 disk.drbd2165.out‚0 network.recv‚619134 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 5.99609375 disk.sda1.in‚0 disk.sda.out‚ 72.5390625& disk.nvme0c0n1.out‚8063.5380859375) disk./.usagePercent‚36.63240249881377ÂÎ 2ß - _wandbruntime‚2235  _runtime‚22355¥[Èà  ' lr-AdamW/pg1‚0.00045813928328925776 ' lr-AdamW/pg2‚0.00045813928328925776 trainer/global_step‚2599 $ _timestamp‚1.784275042898828e+09 _runtime‚2235.027510074 _step‚210;Íá Ç $ _timestamp‚1.784275042898828e+09 _runtime‚2235.027510074 _step‚210 ' lr-AdamW/pg1‚0.00045813928328925776 ' lr-AdamW/pg2‚0.00045813928328925776 trainer/global_step‚2599Ò†Xös2â - _wandbruntime‚2235  _runtime‚2235#»ÎÜ'ã ÖO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10142013 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10142271 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397183415582720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20571058 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24050241 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397183057204224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394439900889088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2743497654272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395154316829220 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392414992928768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2743156315136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20571058 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24060218 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2743497654272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20571056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2773368464384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87134223648768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚2235.028546741 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395153959239970 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397183057204224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5584270 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2599 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87134205036544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24050241 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394439917928448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5584015 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2773452847104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2739306861092 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4558001 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89907573500928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392415009968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10142271 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397183415582720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20571058 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24050241 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397183057204224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394439900889088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2743497654272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395154316829220 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392414992928768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2743156315136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20571058 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24060218 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2743497654272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20571056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2773368464384 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87134223648768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚2235.028546741 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395153959239970 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397183057204224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5584270 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2599 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87134205036544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24050241 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394439917928448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5584015 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2773452847104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2739306861092 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4558001 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89907573500928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚392415009968128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44637981 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394587141918720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2744510288384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24059293 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395301480965564 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392561520640000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10145763 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87167633795072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2744510288384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20578690 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24059293 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397331310860800 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774393058816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397331669246464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394587141918720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2744168942080 trainer/global_step‚2599 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2740317914926 % _timestamp‚1.7842750435172641e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24069284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44637981 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2744168942080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20578690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5586020 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774479531520 P_ë&ç åM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚2235.643077292 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20578688 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89942131938816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20578688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392561503600640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2739977364924 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚397331669246464 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44637981 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚394587141918720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2744510288384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24059293 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395301480965564 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392561520640000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10145763 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87167633795072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2744510288384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20578690 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24059293 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397331310860800 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774393058816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397331669246464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394587141918720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2744168942080 trainer/global_step‚2599 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2740317914926 % _timestamp‚1.7842750435172641e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24069284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44637981 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2744168942080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20578690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5586020 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774479531520 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404595804187648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2275.323383887 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2794193156096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20952658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20952656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88805661575168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402530109320478 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2789923348000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10333245 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚404596162566144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88805642962944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚)üÁÝŽ 1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91630385042944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399740526522368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45457349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚401801952370688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚401801969410048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2793851816960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4645248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404595804187648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2275.323383887 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2794193156096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20952658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20952656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88805661575168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402530109320478 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2789923348000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10333245 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚404596162566144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88805642962944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91630385042944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399740526522368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45457349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚401801952370688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚401801969410048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2793851816960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4645248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401949193400320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24513745 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10336801 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2825852366848 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5690077 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91664831324160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10337073 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401949210439680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2795205790208 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404744057844224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88839090333696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20960290 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4646996 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20960290 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2794864443904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚2275.936477752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _step‚216 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88839071721472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401949210439680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2649 % _timestamp‚1.7842750838106103e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚404744416229888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401949193400320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2794864443904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399887054233600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24513745 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402677988635434 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399887037194240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20960288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402677631046072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2790934401834¹ hNV%÷ åM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399887054233600 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24513745 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402677988635434 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399887037194240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20960288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402677631046072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2790934401834 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5689808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24523736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401949193400320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24513745 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10336801 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2825852366848 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5690077 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91664831324160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10337073 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401949210439680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2795205790208 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404744057844224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88839090333696 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20960290 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4646996 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20960290 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2794864443904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚2275.936477752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _step‚216 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88839071721472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401949210439680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2649 % _timestamp‚1.7842750838106103e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚404744416229888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401949193400320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2794864443904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_byte˜ï¤*•s.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0Ø=púù2ø - _wandbruntime‚2275  _runtime‚2275¯a<ù € %loss/total_loss‚9.420865491227351 2loss/SRqqt1/assignment_loss‚0.5289734779129122 1loss/SRqqt1/detection_loss‚1.0740845345416183 0loss/FBt1/detection_loss‚0.40937187989316076 0loss/FRt2/assignment_loss‚1.1567851481572637 2loss/FBt2/assignment_loss‚0.052742871384656324 /loss/FBt2/detection_loss‚0.7876922297046373 epoch‚6 trainer/global_step‚2649 _runtime‚2275.937416003 _step‚217 /loss/FRt1/assignment_loss‚2.046243882398111 2loss/SRqqt2/assignment_loss‚0.6461042317343937 1loss/FBt1/assignment_loss‚0.02737293594598386 /loss/FRt1/detection_loss‚0.5772956967518535 .loss/FRt2/detection_loss‚1.397207900424892 $ _timestamp‚1.784275083811951e+09 1loss/SRqqt2/detection_loss‚0.7169907023778691È ?ž ú … 0loss/FRt2/assignment_loss‚1.1567851481572637 2loss/FBt2/assignment_loss‚0.052742871384656324 /loss/FBt2/detection_loss‚0.7876922297046373 epoch‚6 trainer/global_step‚2649 _runtime‚2275.937416003 _step‚217 /loss/FRt1/assignment_loss‚2.046243882398111 2loss/SRqqt2/assignment_loss‚0.6461042317343937 1loss/FBt1/assignment_loss‚0.02737293594598386 /loss/FRt1/detection_loss‚0.5772956967518535 .loss/FRt2/detection_loss‚1.397207900424892 $ _timestamp‚1.784275083811951e+09 1loss/SRqqt2/detection_loss‚0.7169907023778691 %loss/total_loss‚9.420865491227351 2loss/SRqqt1/assignment_loss‚0.5289734779129122 1loss/SRqqt1/detection_loss‚1.0740845345416183 0loss/FBt1/detection_loss‚0.40937187989316076Ùçn—#žû ‚(:“ ÁçÒ囫¤ gpu.0.gpu‚90.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Õ¨Û ü ‚(:€ ÁçÒÌõ¨¦# disk.nvme0n1.out‚8087.580078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.988754272460938 cpu‚26.123332465471673 disk.sda3.out‚ 73.5234375 disk.nvme0n1.in‚ 146.05859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 network.sent‚ 4.076923e+06 network.recv‚630389) disk./.usagePercent‚36.63239353855007 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0$ disk./.usageGB‚545.8491363525391 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 73.5234375 disk.dm-0.in‚ 146.05859375" disk.nvme0c0n1.in‚ 146.05859375% disk.nvme0c0n1.out‚8087.580078125( proc.memory.availableMB‚ 495.16796875 disk.dm-0.out‚8087.580078125$ memory_percent‚96.97773456573486 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 6.18359375 disk.rbd0.out‚0 disk.sda.out‚ 73.5234375 disk.rbd2.in‚0 disk.rbd2.out‚0 proc.memory.rssMB‚ 2783.4375 disk.rbd1.in‚0 disk.rbd3.in‚0ä=<“’ý ‚(:‡ žÁçÒ¤ÇĤ gpu.0.gpu‚34.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ýþ ‚(:÷ žÁçÒœÔȦ disk.sda1.in‚0" disk.nvme0c0n1.in‚ 146.05859375 disk.drbd2165.out‚0) disk./.usagePercent‚36.63239328254253 disk.nvme0n1.in‚ 146.05859375 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 6.18359375 disk.dm-0.out‚ 8092.8984375# disk.nvme0c0n1.out‚ 8092.8984375( proc.memory.availableMB‚ 494.27734375 disk.dm-0.in‚ 146.05859375 disk.rbd2.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 73.734375 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.nvme0n1.out‚ 8092.8984375 disk.sda.out‚ 73.734375 disk.sda2.out‚0 disk.rbd1.in‚0 network.recv‚634996$ disk./.usageGB‚545.8491325378418 disk.sda2.in‚0 disk.sda3.out‚ 73.734375! proc.memory.rssMB‚ 2784.109375 disk.sda1.out‚0 disk.rbd3.in‚0* proc.memory.percent‚16.992855072021484 cpu‚26.1233906798298 network.sent‚ 4.139896e+06 disk.drbd2165.in‚0 disk.rbd0.in‚0$ memory_percent‚96.98317050933838$d†ïÿ ‚(:’ ­ÁçÒÒ¹ñ¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒNã € ‚(:‚ ­ÁçÒõ✧( proc.memory.availableMB‚ 494.74609375 network.sent‚ 4.147071e+06! disk.nvme0c0n1.in‚ 146.7734375 disk.rbd3.out‚0 disk.nvme0n1.in‚ 146.7734375$ disk.nvme0n1.out‚8100.8154296875) disk./.usagePercent‚36.63238841839938 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0! disk.dm-0.out‚8100.8154296875 proc.memory.rssMB‚ 2783.4375& disk.nvme0c0n1.out‚8100.8154296875 disk.rbd2.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.988754272460938 disk.sda3.out‚ 73.98046875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 73.98046875 disk.drbd2165.out‚0 cpu‚26.12089204219255 network.recv‚638824$ disk./.usageGB‚545.8490600585938 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 146.7734375 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 73.98046875 disk.rbd1.in‚0 disk.rbd1.out‚ 6.30859375 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚96.98030948638916b¬2 - _wandbruntime‚2315  _runtime‚2315ðž²ÿÇ‚ Á & lr-AdamW/pg1‚0.0004568031436559754 & lr-AdamW/pg2‚0.0004568031436559754 trainer/global_step‚2699 % _timestamp‚1.7842751232996387e+09 _runtime‚2315.425192198 _step‚218B0(‡Ìƒ Æ % _timestamp‚1.7842751232996387e+09 _runtime‚2315.425192198 _step‚218 & lr-AdamW/pg1‚0.0004568031436559754 & lr-AdamW/pg2‚0.0004568031436559754 trainer/global_step‚2699Úþ˜«s2„ - _wandbruntime‚2315  _runtime‚2315E¶F€Û'… ÕO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842751233007498e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93353149074944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2844547318784 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93353043982848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412008551171072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21334258 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409164003852288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5791254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2876049573376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412008909549568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚219 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21334258 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24959145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2875963093504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409906259400986 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412008909549568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2844888657920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409164020891648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10524026 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4732501 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24969122 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4732498 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2840199285018 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21334256 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412008551171072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10523752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24959145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2840539834908 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407066060115968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2699 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5791525 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90477099501568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2844888657920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2844547318784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24969122 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409164020891648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0‚9Pëà'† ÚO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24969122 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409164020891648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842751233007498e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93353149074944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2844547318784 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93353043982848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412008551171072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21334258 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409164003852288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5791254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2876049573376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412008909549568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚219 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21334258 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24959145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2875963093504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409906259400986 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412008909549568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2844888657920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409164020891648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10524026 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4732501 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24969122 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4732498 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2840199285018 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21334256 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412008551171072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10523752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24959145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2840539834908 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407066060115968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2699 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5791525 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90477099501568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2844888657920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2844547318784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508Ûúó’2‡ - _wandbruntime‚2316  _runtime‚2316ñÈi'߈ àM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24978188 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93387488521216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚220 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2845901292032 % _timestamp‚1.7842751239138765e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2845901292032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2876978873344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24968197 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412157163213312 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409311261921280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚2316.039727537 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24968197 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10527816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409311244881920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2845559945728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2845559945728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21341890 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407212587827200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407212570787840 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21341888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90510528260096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10527554 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 trainer/global_step‚2699 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410054138715942 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21341890 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409311244881920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2841210338740 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409311261921280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407212587827200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407212570787840 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21341888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90510528260096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10527554 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 trainer/global_step‚2699 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410054138715942 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21341890 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409311244881920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2841210338740 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409311261921280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24978188 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93387488521216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚220 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2845901292032 % _timestamp‚1.7842751239138765e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2845901292032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2876978873344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24968197 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412157163213312 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409311261921280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚2316.039727537ÜxÞq2Š - _wandbruntime‚2316  _runtime‚2316@cÇú‹ € 1loss/SRqqt2/detection_loss‚0.7166950069330928 /loss/FBt1/detection_loss‚0.5800440347278428 % _timestamp‚1.7842751239160113e+09 0loss/FRt1/assignment_loss‚1.9676631941146974 0loss/FRt2/assignment_loss‚1.0959165791295609 2loss/SRqqt1/assignment_loss‚0.7656714648831014 1loss/FBt2/assignment_loss‚0.02812182589723386 .loss/FRt2/detection_loss‚1.407429378934837 1loss/SRqqt1/detection_loss‚1.1208666372340585 epoch‚6 2loss/SRqqt2/assignment_loss‚0.5973719937461456 1loss/FBt1/assignment_loss‚0.04678891935151441 /loss/FRt1/detection_loss‚0.6365942749596669 /loss/FBt2/detection_loss‚1.0096040109908857 %loss/total_loss‚9.972767320902637 trainer/global_step‚2699 _runtime‚2316.041345342 _step‚221%Ê7" Œ … 1loss/SRqqt2/detection_loss‚0.7166950069330928 /loss/FBt1/detection_loss‚0.5800440347278428 % _timestamp‚1.7842751239160113e+09 0loss/FRt1/assignment_loss‚1.9676631941146974 0loss/FRt2/assignment_loss‚1.0959165791295609 2loss/SRqqt1/assignment_loss‚0.7656714648831014 1loss/FBt2/assignment_loss‚0.02812182589723386 .loss/FRt2/detection_loss‚1.407429378934837 1loss/SRqqt1/detection_loss‚1.1208666372340585 epoch‚6 2loss/SRqqt2/assignment_loss‚0.5973719937461456 1loss/FBt1/assignment_loss‚0.04678891935151441 /loss/FRt1/detection_loss‚0.6365942749596669 /loss/FBt2/detection_loss‚1.0096040109908857 %loss/total_loss‚9.972767320902637 trainer/global_step‚2699 _runtime‚2316.041345342 _step‚221Ý÷xáé“ ‚(:ˆ ¼ÁçÒ­œÎ¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p®È&Ž ‚(:ú ¼ÁçÒäðÞ§ disk.rbd1.out‚ 6.30859375 disk.dm-0.in‚ 147.6015625* proc.memory.percent‚16.992855072021484! disk.nvme0c0n1.in‚ 147.6015625! proc.memory.rssMB‚ 2784.109375 proc.cpu.threads‚17 disk.rbd1.in‚0$ memory_percent‚96.98333740234375 network.recv‚642137) disk./.usagePercent‚36.63238918642198 disk.sda2.in‚0 disk.rbd3.in‚0! disk.dm-0.out‚8111.0048828125" proc.memory.availableMB‚494.25 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd3.out‚0 cpu‚26.11832172475754 disk.nvme0n1.in‚ 147.6015625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚8111.0048828125 disk.sda.out‚ 74.1328125$ disk./.usageGB‚545.8490715026855 disk.rbd2.in‚0 disk.rbd2.out‚0$ disk.nvme0n1.out‚8111.0048828125 disk.sda2.out‚0 disk.dm-1.out‚ 74.1328125 disk.sda1.out‚0 disk.dm-1.in‚0 network.sent‚ 4.16303e+06 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 74.1328125š)§ž ‚(:“ ËÁçÒÛ͵¥ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09;ü ‚(:… ËÁçÒ«ðৠdisk.dm-1.in‚0 network.recv‚646688 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚8122.05322265625 disk.sda3.out‚ 74.3828125 network.sent‚ 4.215059e+06 disk.nvme0n1.in‚ 148.31640625 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 148.31640625 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 74.3828125 disk.rbd3.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚8122.05322265625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0$ memory_percent‚96.98147773742676 disk.dm-1.out‚ 74.3828125 disk.drbd2165.out‚0 disk.sda3.in‚0) disk./.usagePercent‚36.63238893041445 disk.sda2.out‚0 disk.rbd1.out‚ 6.37109375 cpu‚26.115691078491608 disk.rbd0.out‚0' proc.memory.availableMB‚ 494.5546875 proc.memory.rssMB‚ 2783.4375$ disk./.usageGB‚545.8490676879883 disk.rbd2.in‚0 disk.dm-0.in‚ 148.31640625" disk.dm-0.out‚8122.05322265625* proc.memory.percent‚16.988754272460938ÿow-ž‘ ‚(:“ ÚÁçÒ£’ý£ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚255.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.068599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 xË­’ ‚(:„ ÚÁçÒ”ž¾¥ disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 149.14453125 disk.rbd1.in‚0* proc.memory.percent‚16.992855072021484 disk.rbd3.out‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚8130.7529296875! proc.memory.rssMB‚ 2784.109375 disk.dm-1.in‚0 disk.rbd1.out‚ 6.37109375 disk.dm-0.in‚ 149.14453125$ disk.nvme0n1.out‚8130.7529296875 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 74.5390625 proc.cpu.threads‚17 network.recv‚650114# disk./.usageGB‚545.849063873291 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd2.out‚0 disk.drbd2165.out‚0$ memory_percent‚96.98455333709717 cpu‚26.113145527229435 network.sent‚ 4.220206e+06! disk.dm-0.out‚8130.7529296875 disk.nvme0n1.in‚ 149.14453125 disk.sda1.in‚0 disk.dm-1.out‚ 74.5390625( proc.memory.availableMB‚ 494.05078125) disk./.usagePercent‚36.63238867440691 disk.sda.out‚ 74.5390625 disk.sda2.out‚0ÇDë‡2“ - _wandbruntime‚2355  _runtime‚2355f±ï‰Å” ¿ & lr-AdamW/pg1‚0.0004561138116854571 & lr-AdamW/pg2‚0.0004561138116854571 trainer/global_step‚2749 $ _timestamp‚1.784275163243995e+09 _runtime‚ 2355.36958548 _step‚222ñ}»çÊ• Ä trainer/global_step‚2749 $ _timestamp‚1.784275163243995e+09 _runtime‚ 2355.36958548 _step‚222 & lr-AdamW/pg1‚0.0004561138116854571 & lr-AdamW/pg2‚0.0004561138116854571Þ[Ý@X2– - _wandbruntime‚2355  _runtime‚2355°aÜ'— ÖO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416526072373248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92148537427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4819751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25413597 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2895584159744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2891156321816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95075636408320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5894079 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5893822 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419421298154496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416526055333888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25413597 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21715856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4819748 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2927098980352 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419421298154496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416526055333888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25413597 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21715856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_baHåŒO"tch_start/inactive_split.large_pool.freed‚4819748 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2927098980352 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92148537427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4819751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25413597 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2895584159744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2891156321816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95075636408320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5894079 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154ß¾qÍã2™ - _wandbruntime‚2355  _runtime‚2355¾Éûå&š ßM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417430288796450 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚2749 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784275163857817e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21723490 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419569910196736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25422649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5896135 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4821496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25432640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21723490 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416673296363520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25432640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2896255447552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2896255447552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21723488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95110087194112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2355.983646946 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚416673313402880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5895879 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92181966186496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414538104381440 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896596793856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419569551811072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896596793856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416673296363520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95109980012032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10717372 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4821493 _step‚224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21723488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2928121007616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414538121420800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416673313402880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ê›?Æê&› äM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419569910196736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95110087194112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2355.983646946 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚416673313402880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5895879 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92181966186496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414538104381440 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896596793856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419569551811072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896596793856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416673296363520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95109980012032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10717372 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4821493 _step‚224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21723488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2928121007616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414538121420800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416673313402880 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417430288796450 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚2749 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784275163857817e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21723490 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419569910196736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25422649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5896135 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4821496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25432640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21723490 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416673296363520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25432640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2896255447552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2896255447552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21723488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10904654 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22097458 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93819956742144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424658559562002 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426834045137920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2978199789056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5997906 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423888106815488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426834045137920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421717127303168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4907001 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424658917151252 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2946279661568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47975484 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96798261623296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚227 trainer/global_step‚2799 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2945938322432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423888123854848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5997656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93819975354368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426834403516416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 $ _timestamp‚1.784275202936912e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878026 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906998 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22097456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22097458 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25878026 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868049 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2945938322432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _runtime‚2395.066373089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10904907 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2978286268928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421717144342528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2941432258834 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426834403516416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977—tà'¨ ÚO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96798156531200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10904654 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22097458 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93819956742144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424658559562002 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426834045137920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2978199789056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5997906 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423888106815488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426834045137920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421717127303168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4907001 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424658917151252 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2946279661568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47975484 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96798261623296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚227 trainer/global_step‚2799 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2945938322432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423888123854848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5997656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93819975354368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426834403516416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 $ _timestamp‚1.784275202936912e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878026 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906998 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22097456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22097458 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25878026 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868049 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2945938322432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _runtime‚2395.066373089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10904907 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2978286268928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421717144342528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2941432258834 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426834403516416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438ã¹Q 2© - _wandbruntime‚2395  _runtime‚2395èXæ&ª àM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2979230044160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424806081287596 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424035364884480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424035347845120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424806438876958 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842752035490305e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25877101 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4908743 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25877101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22105090 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4908746 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96832716435456 trainer/global_step‚2799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96832615544832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421863637975040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2946950949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93853385500672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426982298794496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22105088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10908427 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚424035347845120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93853404112896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2979312322560 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25887092 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2947292295680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10908684 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚228 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25887092 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5999684 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚421863655014400 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426982657180160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424035364884480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2946950949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2947292295680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426982657180160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088Ìë~¦º« åM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25887092 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2947292295680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10908684 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚228 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25887092 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5999684 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚421863655014400 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426982657180160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.lÂÒÅ1"arge_pool.allocated‚424035364884480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2946950949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2947292295680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426982657180160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2979230044160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424806081287596 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424035364884480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424035347845120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424806438876958 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842752035490305e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25877101 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4908743 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25877101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22105090 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4908746 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96832716435456 trainer/global_step‚2799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96832615544832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421863637975040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2946950949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93853385500672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426982298794496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22105088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10908427 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚424035347845120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93853404112896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2979312322560ä%ÈiÕ2¬ - _wandbruntime‚2395  _runtime‚2395¡T–­ € 0loss/FRt2/assignment_loss‚1.0581429379393872 2loss/SRqqt2/assignment_loss‚0.5908827614305429 1loss/SRqqt2/detection_loss‚0.6777453799272117 /loss/FBt2/detection_loss‚0.7518103513334479 epoch‚6 trainer/global_step‚2799 _runtime‚2395.677680597 0loss/FRt1/assignment_loss‚1.9782770437430282 1loss/FBt2/assignment_loss‚0.01963265048735983 .loss/FRt2/detection_loss‚1.484718351787979 /loss/FBt1/detection_loss‚0.5267670719371433 % _timestamp‚1.7842752035523517e+09 _step‚229 %loss/total_loss‚9.422577595424542 2loss/SRqqt1/assignment_loss‚0.6474141870397239 2loss/FBt1/assignment_loss‚0.037158930500602746 /loss/FRt1/detection_loss‚0.6123973290477696 0loss/SRqqt1/detection_loss‚1.037630600250345Äþ æ ® … 2loss/SRqqt2/assignment_loss‚0.5908827614305429 1loss/SRqqt2/detection_loss‚0.6777453799272117 /loss/FBt2/detection_loss‚0.7518103513334479 epoch‚6 trainer/global_step‚2799 _runtime‚2395.677680597 0loss/FRt1/assignment_loss‚1.9782770437430282 1loss/FBt2/assignment_loss‚0.01963265048735983 .loss/FRt2/detection_loss‚1.484718351787979 /loss/FBt1/detection_loss‚0.5267670719371433 % _timestamp‚1.7842752035523517e+09 _step‚229 %loss/total_loss‚9.422577595424542 2loss/SRqqt1/assignment_loss‚0.6474141870397239 2loss/FBt1/assignment_loss‚0.037158930500602746 /loss/FRt1/detection_loss‚0.6123973290477696 0loss/SRqqt1/detection_loss‚1.037630600250345 0loss/FRt2/assignment_loss‚1.0581429379393872åé¤:“¯ ‚(:ˆ ‡ÂçÒ”ˆ–¥ gpu.0.gpu‚58.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Mþ° ‚(:ü ‡ÂçÒ‘ãŧ disk.rbd2.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 network.recv‚661434 disk.sda.out‚ 75.27734375 disk.sda2.in‚0 disk.dm-1.in‚0# disk./.usageGB‚545.834114074707 disk.rbd2.in‚0 disk.dm-1.out‚ 75.27734375" disk.dm-0.out‚8148.40283203125$ memory_percent‚96.98288440704346( proc.memory.availableMB‚ 494.32421875 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚150.6875' disk.nvme0c0n1.out‚8148.40283203125 disk.sda3.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚150.6875 disk.sda1.in‚0 disk.sda3.out‚ 75.27734375 proc.memory.rssMB‚ 2783.4375* proc.memory.percent‚16.988754272460938 cpu‚26.11244894914697 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚150.6875 network.sent‚ 4.291178e+06* disk./.usagePercent‚36.631385380880225% disk.nvme0n1.out‚8148.40283203125 disk.rbd1.out‚ 6.49609375 disk.sda.in‚0 disk.rbd1.in‚0QxÁ2± - _wandbruntime‚2412  _runtime‚2412L,‚¿=² ¹{ 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.33590832352638245 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7516413927078247 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2015405148267746 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5482590198516846 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7271358370780945 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6389333009719849 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6652159094810486 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4438420236110687 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 # _timestamp‚1.78427522080755e+09 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909255504608154 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03888474032282829 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +particle/specificity‚0.9123474359512329 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7238410115242004 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7232422828674316 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.994761049747467 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10083924978971481 particle/accuracy_2_of_2‚1 _runtime‚2412.933844805 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5558049082756042 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1809447854757309 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 *jet/accuracy_2_of_4‚0.7627192139625549 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6946403980255127 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 *jet/accuracy_3_of_3‚0.0667596384882927 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6725430488586426 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01384006254374981 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 * Purity/*FRt1SRqqt2FBt/FBt_purity‚0.875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841270446777344 *jet/accuracy_1_of_3‚0.9282978773117065 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971743822097778 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744220972061157 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.77186119556427 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40746229887008667 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7148209810256958 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9937958717346191 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6899515986442566 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9920990467071533 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_3‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.726649820804596 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1678878366947174 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05233224853873253 *jet/accuracy_2_of_3‚0.5489223003387451 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9956160187721252 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960300922393799 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.666893482208252 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9934346079826355 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4643711745738983 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8115307688713074 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444477558136 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2683699131011963 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9158798456192017 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6274518966674805 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0509839728474617 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6877224445343018 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20630986988544464 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2708333134651184 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0467676967382431 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.134635791182518 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960300922393799 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18797338008880615 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 'particle/f_score‚0.6245429515838623 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9923520088195801 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9831351041793823 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9159548878669739 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17575429379940033 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6821915507316589 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04920404776930809 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5295833349227905 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2201704978942871 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19819363951683044 6validation_average_jet_accuracy‚0.3772626519203186 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4207031726837158 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.791364312171936 _step‚230 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5303004384040833 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4566454589366913 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.853303074836731 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6603693962097168 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0467676967382431 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11759389936923981 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17515788972377777 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28451526165008545 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 epoch‚6 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32060036063194275 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7760394215583801 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18893374502658844 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20630986988544464 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2377287745475769 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21567237377166748 +jet/accuracy_3_of_4‚0.08303418755531311 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03908340632915497 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9851487278938293 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9952360987663269 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7680650353431702 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.801662266254425 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6673010587692261 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05265245959162712 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7938586473464966 *jet/accuracy_1_of_2‚0.8018171191215515 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19786852598190308 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4367261528968811 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 +particle/sensitivity‚0.5996180772781372 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 trainer/global_step‚2799 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892666935920715 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05125812068581581 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.8066841959953308 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9833630919456482 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2188212126493454 +jet/accuracy_2_of_2‚0.14726422727108002 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8282830119132996 /particle/accuracy_3_of_4‚0.9946429133415222 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21667273342609406 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9828922748565674 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6797816157341003 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1968676745891571 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7787559032440186 *jet/accuracy_1_of_4‚0.9598299860954285 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6851990222930908 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28564244508743286 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8783000111579895 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4072706997394562 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4741699695587158 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6446971297264099 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7688428163528442 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5641884803771973 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7466275691986084 )loss/val_total_loss‚9.275250834072407 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7206249833106995 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7386813163757324 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 'particle/accuracy‚0.845183789730072 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19853344559669495 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444477558136 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 jet/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7024187445640564 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020828112959861755 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21861660480499268 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7015567421913147 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7760367393493652 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7293179631233215 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872109293937683 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6899515986442566 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15832997858524323 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025502726435661316 jet/accuracy_3_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8142495155334473 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8559584617614746 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11740907281637192 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8282830119132996 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7866969108581543 6"Purity/0FRt*SRqqt2FBt/event_purity‚0.9174844622612 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7118188142776489 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20567195117473602 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612ég°\³ ¾{ 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444477558136 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2683699131011963 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9158798456192017 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6274518966674805 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0509839728474617 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6877224445343018 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20630986988544464 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2708333134651184 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0467676967382431 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.134635791182518 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960300922393799 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18797338008880615 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 'particle/f_score‚0.6245429515838623 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9923520088195801 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9831351041793823 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9159548878669739 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17575429379940033 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6821915507316589 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04920404776930809 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5295833349227905 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2201704978942871 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19819363951683044 6validation_average_jet_accuracy‚0.3772626519203186 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4207031726837158 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.791364312171936 _step‚230 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5303004384040833 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4566454589366913 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.853303074836731 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6603693962097168 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0467676967382431 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11759389936923981 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17515788972377777 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28451526165008545 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 epoch‚6 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32060036063194275 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7760394215583801 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18893374502658844 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20630986988544464 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2377287745475769 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21567237377166748 +jet/accuracy_3_of_4‚0.08303418755531311 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03908340632915497 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9851487278938293 7 Purity/1FRt*SRqqt*FBt/ÖEoš*FBt_purity‚0.9952360987663269 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7680650353431702 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.801662266254425 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6673010587692261 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05265245959162712 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7938586473464966 *jet/accuracy_1_of_2‚0.8018171191215515 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19786852598190308 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4367261528968811 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 +particle/sensitivity‚0.5996180772781372 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927884936332703 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 trainer/global_step‚2799 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892666935920715 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05125812068581581 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.8066841959953308 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9833630919456482 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2188212126493454 +jet/accuracy_2_of_2‚0.14726422727108002 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8282830119132996 /particle/accuracy_3_of_4‚0.9946429133415222 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21667273342609406 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9828922748565674 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6797816157341003 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1968676745891571 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7787559032440186 *jet/accuracy_1_of_4‚0.9598299860954285 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6851990222930908 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28564244508743286 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8783000111579895 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4072706997394562 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4741699695587158 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6446971297264099 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7688428163528442 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5641884803771973 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7466275691986084 )loss/val_total_loss‚9.275250834072407 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7206249833106995 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7386813163757324 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 'particle/accuracy‚0.845183789730072 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19853344559669495 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444477558136 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 jet/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7024187445640564 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020828112959861755 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21861660480499268 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7015567421913147 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7760367393493652 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7293179631233215 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872109293937683 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6899515986442566 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15832997858524323 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025502726435661316 jet/accuracy_3_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8142495155334473 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8559584617614746 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11740907281637192 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8282830119132996 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7866969108581543 6"Purity/0FRt*SRqqt2FBt/event_purity‚0.9174844622612 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7118188142776489 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20567195117473602 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.33590832352638245 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7516413927078247 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2015405148267746 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5482590198516846 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7271358370780945 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6389333009719849 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6652159094810486 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4438420236110687 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 # _timestamp‚1.78427522080755e+09 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909255504608154 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03888474032282829 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +particle/specificity‚0.9123474359512329 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7238410115242004 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7232422828674316 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.994761049747467 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10083924978971481 particle/accuracy_2_of_2‚1 _runtime‚2412.933844805 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5558049082756042 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1809447854757309 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 *jet/accuracy_2_of_4‚0.7627192139625549 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6946403980255127 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 *jet/accuracy_3_of_3‚0.0667596384882927 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6725430488586426 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01384006254374981 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 * Purity/*FRt1SRqqt2FBt/FBt_purity‚0.875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841270446777344 *jet/accuracy_1_of_3‚0.9282978773117065 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971743822097778 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744220972061157 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.77186119556427 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40746229887008667 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7148209810256958 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9937958717346191 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6899515986442566 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9920990467071533 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_3‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.726649820804596 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1678878366947174 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05233224853873253 *jet/accuracy_2_of_3‚0.5489223003387451 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9956160187721252 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960300922393799 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9889224171638489 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.666893482208252 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9934346079826355 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4643711745738983 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8115307688713074æ% tˆ›´ ‚(: –ÂçÒñ¨¶¤ gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.259' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.251800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0smÛµ ‚(:ý –ÂçÒñ뢥 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.dm-0.out‚ 8206.3671875$ memory_percent‚98.46792221069336 network.sent‚ 4.353984e+06# disk./.usageGB‚546.085880279541 disk.rbd0.out‚0) proc.memory.percent‚18.48738193511963# disk.nvme0c0n1.out‚ 8219.6171875 disk.sda3.in‚0 disk.sda3.out‚ 75.45703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚26.127460361182028 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 6.49609375 disk.rbd3.out‚0 disk.dm-0.in‚ 151.40234375# proc.memory.rssMB‚ 3028.97265625 network.recv‚665029) disk./.usagePercent‚36.64828162213255 disk.dm-1.out‚ 75.45703125 disk.nvme0n1.in‚ 151.40234375 disk.sda.out‚ 75.45703125" disk.nvme0c0n1.in‚ 151.40234375 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚0! disk.nvme0n1.out‚ 8219.6171875 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 251.015625VÚc‘¶ ‚(:† ¥ÂçÒÈ÷î¤ gpu.0.gpu‚81.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05Ø|n· ‚(:÷ ¥ÂçÒ»„©§ disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda3.out‚75.875% disk.nvme0n1.out‚8411.26123046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 6.55859375 disk.rbd3.out‚0" memory_percent‚95.473313331604( proc.memory.availableMB‚ 741.65234375 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 152.23046875 disk.sda2.in‚0 cpu‚26.122300594919 proc.cpu.threads‚17 network.recv‚669319$ disk./.usageGB‚545.8349304199219' disk.nvme0c0n1.out‚8411.26123046875 disk.sda.out‚75.875 disk.sda2.out‚0! proc.memory.rssMB‚ 2784.109375* proc.memory.percent‚16.992855072021484 network.sent‚ 4.412662e+06 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 152.23046875 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 152.23046875" disk.dm-0.out‚8411.26123046875 disk.dm-1.in‚0) disk./.usagePercent‚36.63144016649255 disk.sda.in‚0 disk.dm-1.out‚75.875 disk.rbd1.in‚0^py“¸ ‚(:ˆ ´ÂçÒƒ ‡¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ñi„¹ ‚(:ý ´ÂçÒ÷—§ disk.sda2.in‚0 disk.dm-1.out‚76.0625 disk.rbd2.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 152.23046875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 741.27734375* proc.memory.percent‚16.992878913879395& disk.nvme0c0n1.out‚8663.2666015625 disk.sda.out‚76.0625! disk.dm-0.out‚8663.2666015625 disk.rbd2.in‚0$ memory_percent‚95.47560214996338 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.out‚76.0625 disk.dm-0.in‚ 152.23046875$ disk.nvme0n1.out‚8663.2666015625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2784.11328125 cpu‚26.119659731571264 disk.rbd3.out‚0 network.sent‚ 4.417921e+06 network.recv‚672879 disk.rbd1.out‚ 6.55859375 disk.nvme0n1.in‚ 152.23046875) disk./.usagePercent‚36.63134774777269# disk./.usageGB‚545.833553314209 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.in‚0d›¬12º - _wandbruntime‚2454  _runtime‚2454¯à—È»  _runtime‚2454.674337688 _step‚231 ' lr-AdamW/pg1‚0.00045469284480634454 ' lr-AdamW/pg2‚0.00045469284480634454 trainer/global_step‚2849 $ _timestamp‚1.784275262548732e+096±¯ºÍ¼ Ç ' lr-AdamW/pg1‚0.00045469284480634454 ' lr-AdamW/pg2‚0.00045469284480634454 trainer/global_step‚2849 $ _timestamp‚1.784275262548732e+09 _runtime‚2454.674337688 _step‚231çŸZÆ+2½ - _wandbruntime‚2454  _runtime‚2454r—f Ú'¾ ÔO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433961092581824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4996771 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3009768470976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002529 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11129614 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11129358 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22599722 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433179741986816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433179724947456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26402805 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6132843 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3047559892480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433961450171074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436194091979776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599722 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22599724 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433179724947456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4996768 % _timestamp‚1.7842752625497904e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430951324110848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430951341150208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3010109020866 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚436194450358272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436194091979776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26402805 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26392828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98554885273088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95507306768384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3014708371456 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48992550 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436194450358272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26392828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98554885273088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95507306768384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3014708371456 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48992550 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436194450358272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4996771 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3009768470976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002529 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11129614 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11129358 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22599722 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433179741986816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433179724947456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26402805 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6132843 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3047559892480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433961450171074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436194091979776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599722 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22599724 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433179724947456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4996768 % _timestamp‚1.7842752625497904e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430951324110848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚430951341150208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3010109020866 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚436194450358272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436194091979776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26402805 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169è<¼7:2À - _wandbruntime‚2455  _runtime‚24557ÎRå&Á ßM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚2849 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3015721005568 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4998513 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚436342345636352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚436342704022016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26401880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433326965977088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11133434 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98589237715968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26411871 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436342704022016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22607354 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3015379659264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6134918 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11133184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4998516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431097851822080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3010779524698 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431097834782720 $ _timestamp‚1.784275263162899e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009234 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3015721005568 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95540735526912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26411871 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¹R½êê& äM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3011120074700 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431097851822080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3010779524698 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431097834782720 $ _timestamp‚1.784275263162899e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009234 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3015721005568 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95540735526912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26411871 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚2849 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3015721005568 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4998513 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚436342345636352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚436342704022016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26401880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433326965977088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11133434 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98589237715968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26411871 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436342704022016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22607354 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3015379659264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6134918 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11133184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4998516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443606838963200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3098820685824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441337242662332 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 # _timestamp‚1.78427530233434e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5084018 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440541776429056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97178763307008 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97178744694784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2899 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26857257 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚236 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26857257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3098911360000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441337600251582 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0‡vá'Ñ ÛO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26857257 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚236 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26857257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3098911360000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441337600251582 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22981324 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443607197341696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438276857704448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22981322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838581 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚2494.460102557 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3065062534144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3060725507774 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100277674667008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443606838963200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3065062534144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6236609 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22981322 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11320630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11320349 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440541793468416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26847280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6236331 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443606838963200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3098820685824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441337242662332 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 # _timestamp‚1.78427530233434e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5084018 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚440541776429056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97178763307008 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97178744694784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2899 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ìçz÷ü2Ò - _wandbruntime‚2495  _runtime‚2495Êïä6è&Ó âM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11324101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100312122253824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438423368376320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443755451005440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26856332 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49845286 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5085766 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443755092619776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26856332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3066075161088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11324351 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97212173453312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22988956 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97212192065536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚237 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3066416507392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3099843715584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438423385415680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443755451005440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26866323 trainer/global_step‚2899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5085763 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842753029469552e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49845286 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22988954 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26866323 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6238585 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3061396011606 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22988956 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441485121977288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440689034498048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3066075161088_û Ô çM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚237 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3066416507392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3099843715584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438423385415680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443755451005440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26866323 trainer/global_step‚2899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5085763 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842753029469552e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49845286 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22988954 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26866323 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6238585 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3061396011606 KEDeviceStatsMonitor.on_train_batchl ]_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22988956 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441485121977288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440689034498048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3066075161088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11324101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100312122253824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438423368376320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443755451005440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26856332 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49845286 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5085766 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443755092619776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26856332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3066075161088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11324351 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97212173453312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22988956 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97212192065536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0í»æ2Õ - _wandbruntime‚2495  _runtime‚2495nV’ˆÖ ý /loss/FRt1/assignment_loss‚2.029812015659575 2loss/SRqqt1/assignment_loss‚0.5725691616483296 1loss/FBt2/assignment_loss‚0.06032532451814253 1loss/SRqqt1/detection_loss‚1.0602052680093996 epoch‚7 trainer/global_step‚2899 2loss/FBt1/assignment_loss‚0.029239582169540517 1loss/SRqqt2/detection_loss‚0.7037351694063386 0loss/FRt2/assignment_loss‚1.2009078719725845 /loss/FRt1/detection_loss‚0.5680117114616853 /loss/FRt2/detection_loss‚1.3609169597765562 /loss/FBt1/detection_loss‚0.5240153197851711 /loss/FBt2/detection_loss‚0.7551332702239354 $ _timestamp‚1.784275302949962e+09 2loss/SRqqt2/assignment_loss‚0.6174946098227667 %loss/total_loss‚9.482366264454026 _runtime‚ 2495.075297 _step‚238a“´× ‚ 0loss/FRt2/assignment_loss‚1.2009078719725845 /loss/FRt1/detection_loss‚0.5680117114616853 /loss/FRt2/detection_loss‚1.3609169597765562 /loss/FBt1/detection_loss‚0.5240153197851711 /loss/FBt2/detection_loss‚0.7551332702239354 $ _timestamp‚1.784275302949962e+09 2loss/SRqqt2/assignment_loss‚0.6174946098227667 %loss/total_loss‚9.482366264454026 _runtime‚ 2495.075297 _step‚238 /loss/FRt1/assignment_loss‚2.029812015659575 2loss/SRqqt1/assignment_loss‚0.5725691616483296 1loss/FBt2/assignment_loss‚0.06032532451814253 1loss/SRqqt1/detection_loss‚1.0602052680093996 epoch‚7 trainer/global_step‚2899 2loss/FBt1/assignment_loss‚0.029239582169540517 1loss/SRqqt2/detection_loss‚0.7037351694063386îŸvØ ‚(:ˆ ðÂçÒ°³¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 yCÙ ‚(:„ ðÂçÒùʧ disk.sda.out‚ 77.640625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0$ memory_percent‚95.47855854034424 cpu‚26.110367043204732 disk.drbd2165.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 154.94140625$ disk.nvme0n1.out‚8696.0966796875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 6.74609375 network.sent‚ 4.506944e+06 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 740.79296875$ disk./.usageGB‚545.8357963562012 disk.sda1.out‚0 disk.sda3.out‚ 77.640625 disk.rbd3.in‚0) disk./.usagePercent‚36.63149828020283 disk.rbd2.out‚0# proc.memory.rssMB‚ 2784.11328125* proc.memory.percent‚16.992878913879395 proc.cpu.threads‚17 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 154.94140625 disk.sda.in‚0 disk.dm-1.out‚ 77.640625 network.recv‚687370 disk.dm-0.in‚ 154.94140625! disk.dm-0.out‚8696.0966796875& disk.nvme0c0n1.out‚8696.0966796875w‰6øÚ ‚(:’ ÿÂçÒðêÕ¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û ‚(:‹ ÿÂçÒ—Ùñ¦' disk.nvme0c0n1.out‚8705.43994140625 disk.rbd1.in‚0 disk.dm-0.in‚ 155.31640625 disk.sda1.in‚0* proc.memory.percent‚16.992878913879395 cpu‚26.10808012246851 disk.dm-1.in‚0 disk.rbd1.out‚ 6.80859375 disk.sda2.out‚0# proc.memory.rssMB‚ 2784.11328125 disk.drbd2165.out‚0$ memory_percent‚95.47810554504395' proc.memory.availableMB‚ 740.8671875 proc.cpu.threads‚17 network.sent‚ 4.556858e+06 disk.drbd2165.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚36.63152720905419$ disk./.usageGB‚545.8362274169922 disk.rbd3.in‚0 disk.rbd2.in‚0% disk.nvme0n1.out‚8705.43994140625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 155.31640625 disk.sda.out‚ 77.87109375 network.recv‚691934 disk.sda3.in‚0 disk.sda3.out‚ 77.87109375 disk.dm-1.out‚ 77.87109375 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚8705.43994140625 disk.nvme0n1.in‚ 155.31640625·q÷2Ü - _wandbruntime‚2534  _runtime‚2534rÒ»ÇÝ Á & lr-AdamW/pg1‚0.0004532157797332366 & lr-AdamW/pg2‚0.0004532157797332366 trainer/global_step‚2949 % _timestamp‚1.7842753423036535e+09 _runtime‚2534.429311176 _step‚239Ÿû 5ÌÞ Æ & lr-AdamW/pg1‚0.0004532157797332366 & lr-AdamW/pg2‚0.0004532157797332366 trainer/global_step‚2949 % _timestamp‚1.7842753423036535e+09 _runtime‚2534.429311176 _step‚239ïÃkè½2ß - _wandbruntime‚2534  _runtime‚25344J»Þ'à ØO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2949 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3115758035968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362922 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5171268 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚445602408337408 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27311709 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3116099375104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3150089190400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448713750332090 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447903844950016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11511248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11511514 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6340243 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3115758035968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447903827910656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448713392742840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664654 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27301732 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445602391298048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98850182621184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27311709 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3150186156032 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451019944325120 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362924 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98850201233408 % _timestamp‚1.7842753423048117e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362922 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451019585946624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447903827910656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3111341994682 _runtime‚2534.430599038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50674633 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447903844950016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451019585946624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447903827910656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3111341994682 _runtime‚2534.430599038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50674633 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447903844950016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3116099375104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3150089190400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448713750332090 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447903844950016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11511248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11511514 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6340243 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3115758035968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447903827910656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448713392742840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664654 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968ð±ßÉ â ‚(:’ ŽÃçÒ߸´¥ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýMB ã ‚(:þ ŽÃçÒŸ¼„© disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2784.11328125 disk.dm-0.in‚ 156.03125 disk.rbd2.out‚0 network.recv‚694934* disk./.usagePercent‚36.631663917077475 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 cpu‚26.105856115986672 network.sent‚ 4.574932e+06 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 156.03125 disk.sda1.out‚0& disk.nvme0c0n1.out‚8715.8955078125 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 740.87109375* proc.memory.percent‚16.992878913879395 proc.cpu.threads‚17# disk./.usageGB‚545.838264465332$ memory_percent‚95.47808170318604 disk.sda3.in‚0 disk.rbd1.out‚ 6.80859375$ disk.nvme0n1.out‚8715.8955078125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 156.03125 disk.drbd2165.out‚0 disk.sda.out‚ 78.0859375 disk.sda2.in‚0 disk.sda3.out‚ 78.0859375 disk.dm-1.out‚ 78.0859375! disk.dm-0.out‚8715.8955078125 disk.rbd2.in‚0¿G62ä - _wandbruntime‚2535  _runtime‚2535¯0èVå âM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451167839603200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27320775 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚448051068940288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11515300 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98883611379712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448051085979648 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50681338 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2949 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3116770662912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11515025 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50681338 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23370554 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98883629991936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27320775 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448051068940288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102034725447168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23370554 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27310784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3117112009216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445748919009280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3117112009216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451168197988864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5173013 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451168197988864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚448860914468434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6342284 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3112012498514 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27310784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23370556 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5173016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269ÏŸåí&æ çM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7842753429243512e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6342012 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3151114067456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3112353048516 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451168197988864 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚448860914468434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6342284 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3112012498514 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27310784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23370556 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5173016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451167839603200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27320775 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚448051068940288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11515300 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98883611379712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448051085979648 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50681338 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2949 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3116770662912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11515025 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50681338 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23370554 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98883629991936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27320775 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448051068940288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102034725447168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23370554 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27310784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3117112009216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445748919009280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3117112009216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451168197988864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5173013 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458432332930048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458432691308544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458432332930048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11702272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3166794876928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456089900412598 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23744522 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27756184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3161617931700 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100521639159808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3201312828416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27766161 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3166453537792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3161958481590 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27756184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3166794876928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6443™*Þ¯‹481 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103722836410368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842753822453349e+09 _step‚244 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452927941931008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51500706 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23744524 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455265896431616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5258518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100521620547584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976w½÷Èå'ó ßO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455265879392256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456089542823348 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11701999 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458432332930048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458432691308544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458432332930048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11702272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3166794876928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456089900412598 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23744522 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27756184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3161617931700 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100521639159808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3201312828416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27766161 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3166453537792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3161958481590 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27756184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3166794876928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6443481 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103722836410368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842753822453349e+09 _step‚244 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452927941931008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51500706 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23744524 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455265896431616 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5258518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100521620547584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455265879392256 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23744522 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458432691308544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ô­ Ê2ô - _wandbruntime‚2574  _runtime‚2574ÂS÷qê&õ äM trainer/global_step‚2999 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3167466164736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6445500 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455413137461248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455413120421888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517390 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3202326520320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5260263 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3167807511040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27775227 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11705763 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453074435563520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458580586586624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11706028 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455413120421888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27765236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458580944972288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚245 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100555049306112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100555067918336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23752156 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527383 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6445762 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23752156 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3162628985422 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103757285159424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27775227 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3167466164736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23752154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103757394438656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3167807511040 % _timestamp‚1.7842753828581402e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456237064548942 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3162969535424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453074452602880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456237422138304 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455413137461248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚2574.983952952 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÿ¬Ùkï&ö éM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103757394438656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3167807511040 % _timestamp‚1.7842753828581402e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456237064548942 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3162969535424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453074452602880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456237422138304 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455413137461248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚2574.983952952 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2999 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3167466164736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6445500 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455413137461248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455413120421888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517390 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3202326520320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5260263 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3167807511040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27775227 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11705763 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453074435563520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458580586586624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11706028 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455413120421888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27765236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458580944972288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚245 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100555049306112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100555067918336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23752156 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527383 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6445762 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23752156 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3162628985422 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103757285159424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27775227 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3167466164736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23752154õÙy”€2÷ - _wandbruntime‚2574  _runtime‚2574arOø  2loss/FBt1/assignment_loss‚0.011324341032005828 1loss/SRqqt2/detection_loss‚0.7064344243489727 /loss/FBt2/detection_loss‚0.8311575229412743 % _timestamp‚1.7842753828603582e+09 1loss/FBt2/assignment_loss‚0.04488350322701048 /loss/FRt1/detection_loss‚0.5632603215425511 %loss/total_loss‚9.110788659460862 _runtime‚2574.985696724 _step‚246 2loss/SRqqt2/assignment_loss‚0.5586315009109448 trainer/global_step‚2999 epoch‚7 /loss/FRt1/assignment_loss‚2.007450058565035 2loss/SRqqt1/assignment_loss‚0.5982910527126029 /loss/FRt2/detection_loss‚1.2528887754271232 1loss/SRqqt1/detection_loss‚0.9180718711983119 /loss/FBt1/detection_loss‚0.4380927545005094 0loss/FRt2/assignment_loss‚1.1803025330545203Pa;>ù † trainer/global_step‚2999 epoch‚7 /loss/FRt1/assignment_loss‚2.007450058565035 2loss/SRqqt1/assignment_loss‚0.5982910527126029 /loss/FRt2/detection_loss‚1.2528887754271232 1loss/SRqqt1/detection_loss‚0.9180718711983119 /loss/FBt1/detection_loöµ×gøss‚0.4380927545005094 0loss/FRt2/assignment_loss‚1.1803025330545203 2loss/FBt1/assignment_loss‚0.011324341032005828 1loss/SRqqt2/detection_loss‚0.7064344243489727 /loss/FBt2/detection_loss‚0.8311575229412743 % _timestamp‚1.7842753828603582e+09 1loss/FBt2/assignment_loss‚0.04488350322701048 /loss/FRt1/detection_loss‚0.5632603215425511 %loss/total_loss‚9.110788659460862 _runtime‚2574.985696724 _step‚246 2loss/SRqqt2/assignment_loss‚0.5586315009109448öCV"´ ú ‚(:€ »ÃçÒà½Õ¦ network.sent‚ 4.684858e+06# disk.nvme0n1.out‚8741.169921875" disk.nvme0c0n1.in‚ 157.57421875 disk.dm-1.in‚0 disk.dm-1.out‚ 80.078125# memory_percent‚95.4789400100708( proc.memory.availableMB‚ 740.73046875$ disk./.usageGB‚545.8441925048828 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 6.93359375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-0.in‚ 157.57421875) disk./.usagePercent‚36.63206175278569 disk.sda1.out‚0% disk.nvme0c0n1.out‚8741.169921875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚706757 disk.rbd2.out‚0 disk.sda3.out‚ 80.078125 disk.rbd1.in‚0* proc.memory.percent‚16.992878913879395 proc.cpu.threads‚17 disk.sda.out‚ 80.078125 disk.sda2.out‚0 cpu‚26.099330338654127 disk.rbd2.in‚0 disk.nvme0n1.in‚ 157.57421875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.dm-0.out‚8741.169921875# proc.memory.rssMB‚ 2784.11328125M‹¼z‘û ‚(:† »ÃçÒÊš£¥ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃD—Nžü ‚(:“ ÊÃçÒðµ¥ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐH¨ù ý ‚(:€ ÊÃçÒÉ⼧ disk.rbd3.in‚0$ memory_percent‚95.48006057739258 proc.cpu.threads‚17& disk.nvme0n1.out‚10318.04833984375 disk.drbd2165.in‚0 disk.sda2.out‚0 network.sent‚ 4.736911e+06 network.recv‚710810 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚81.125# disk.dm-0.out‚10318.04833984375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 158.40234375 disk.sda2.in‚0 disk.dm-0.in‚ 158.40234375* disk./.usagePercent‚36.808198776456024$ disk./.usageGB‚548.4687614440918 disk.nvme0n1.in‚ 158.40234375( disk.nvme0c0n1.out‚10318.04833984375 disk.sda.out‚81.125 disk.dm-1.in‚0 disk.rbd1.out‚ 6.93359375* proc.memory.percent‚16.992878913879395 disk.rbd3.out‚0# proc.memory.rssMB‚ 2784.11328125 cpu‚26.096950359310814 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 740.546875 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-1.out‚81.125^ˆMžþ ‚(:“ ÙÃçÒã£Ð¢ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚247.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.404199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýkÒ, ÿ ‚(:€ ÙÃçÒ‘Œñ¥ disk.rbd0.out‚0 disk.rbd2.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚95.47977447509766% proc.memory.availableMB‚ 740.59375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 159.1171875* disk./.usagePercent‚36.631981622427475! disk.nvme0c0n1.in‚ 159.1171875 disk.dm-1.out‚ 81.546875 disk.rbd1.out‚ 6.99609375 disk.rbd0.in‚0 disk.sda.out‚ 81.546875 disk.dm-1.in‚0 disk.drbd2165.in‚0 cpu‚26.09474104440011" disk.dm-0.out‚11454.0205078125% disk.nvme0n1.out‚11454.0205078125 proc.cpu.threads‚17$ disk./.usageGB‚545.8429985046387 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.992878913879395 disk.sda3.out‚ 81.546875 disk.rbd1.in‚0 disk.dm-0.in‚ 159.1171875 disk.rbd2.in‚0# proc.memory.rssMB‚ 2784.11328125 network.sent‚ 4.74199e+06 network.recv‚714182' disk.nvme0c0n1.out‚11454.0205078125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0VüËA2€ - _wandbruntime‚2614  _runtime‚2614ç®%ŸÇ Á & lr-AdamW/pg1‚0.0004516829959365583 & lr-AdamW/pg2‚0.0004516829959365583 trainer/global_step‚3049 % _timestamp‚1.7842754219326234e+09 _runtime‚2614.058295016 _step‚247Â`+ÅÌ‚ Æ _step‚247 & lr-AdamW/pg1‚0.0004516829959365583 & lr-AdamW/pg2‚0.0004516829959365583 trainer/global_step‚3049 % _timestamp‚1.7842754219326234e+09 _runtime‚2614.058295016÷"‹R 2ƒ - _wandbruntime‚2614  _runtime‚2614Xèà'„ ÚO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465845079913472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24126124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462627930873856 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6546890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842754219337616e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088¢šËKå'… ßO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11892658 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28220613 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462627947913216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚463465692903856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5345771 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252427496960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11892931 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6547160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3212574968498 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28210636 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465845438291968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3217490378752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105445485970944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465845438291968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465845079913472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3217149039616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842754219337616e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465845079913472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24126124 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462627930873856 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6546890 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6549208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28219688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3253539966976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462775171903488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28229679 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463613572218812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465993691955712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465993333570048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460399986196480 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11896450 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102226505844736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.allÓ’Œ‹& .freed‚463613214629450 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462775188942848 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5347513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24133754 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24133754 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784275422545478e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24133756 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28229679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6548937 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6549208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28219688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3253539966976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462775171903488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28229679 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463613572218812 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465993691955712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465993333570048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460399986196480 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11896450 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102226505844736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463613214629450 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462775188942848 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5347513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24133754 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24133754 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784275422545478e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24133756 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28229679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6548937 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24507724 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28675065 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473257826896896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3263191455406 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3303578340864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470841842984364 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107168175639040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24507724 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28675065 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473257826896896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3263191455406 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3303578340864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470841842984364 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107168175639040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470989364709958 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3269198514688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28674140 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12087903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103897925158912 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473406080553472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5434763 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3304693280256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3264202509240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470137240424448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103897943771136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842754625181737e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _step‚253 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3263861959238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3269198514688 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470137223385088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5434766 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189494 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3268857168384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6653400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199487 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0æ)sï&˜ éM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470989364709958 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3269198514688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28674140 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12087903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103897925158912 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473406080553472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5434763 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3304693280256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3264202509240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470137240424448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103897943771136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842754625181737e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _step‚253 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3263861959238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3269198514688 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470137223385088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5434766 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189494 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3268857168384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6653400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199487 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3099ý™£^2™ - _wandbruntime‚2654  _runtime‚2654D2 »š ‚ 2loss/FBt2/assignment_loss‚0.019898215480030297 /loss/FRt1/detection_loss‚0.5283660315899604 epoch‚7 0loss/FRt1/assignment_loss‚1.9933988081129719 /loss/FRt2/detection_loss‚1.2638131524901837 1loss/SRqqt2/detection_loss‚0.6409062485813569 /loss/FBt1/detection_loss‚0.5251279728832624 0loss/FRt2/assignment_loss‚1.0604623814251113 2loss/SRqqt1/assignment_loss‚0.6284988000051817 1loss/SRqqt1/detection_loss‚0.9297999062202871 /loss/FBt2/detection_loss‚0.7972657960795221 %loss/total_loss‚8.979291451533877 % _timestamp‚1.7842754625226874e+09 _step‚254 2loss/SRqqt2/assignment_loss‚0.5511893060066564 trainer/global_step‚3099 _runtime‚2654.650516415 1loss/FBt1/assignment_loss‚0.04056483265935241¸cÖå › ‡ %loss/total_loss‚8.979291451533877 % _timestamp‚1.7842754625226874e+09 _step‚254 2loss/SRqqt2/assignment_loss‚0.5511893060066564 trainer/global_step‚3099 _runtime‚2654.650516415 1loss/FBt1/assignment_loss‚0.04056483265935241 2loss/FBt2/assignment_loss‚0.019898215480030297 /loss/FRt1/detection_loss‚0.5283660315899604 epoch‚7 0loss/FRt1/assignment_loss‚1.9933988081129719 /loss/FRt2/detection_loss‚1.2638131524901837 1loss/SRqqt2/detection_loss‚0.6409062485813569 /loss/FBt1/detection_loss‚0.5251279728832624 0loss/FRt2/assignment_loss‚1.0604623814251113 2loss/SRqqt1/assignment_loss‚0.6284988000051817 1loss/SRqqt1/detection_loss‚0.9297999062202871 /loss/FBt2/detection_loss‚0.7972657960795221þt•¦œ ‚(:’ †ÄçÒð’Ä£ gpu.0.gpu‚48.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÜH ‚(:Ž †ÄçÒ«¹î¥ disk.rbd2.out‚0& disk.nvme0n1.out‚11487.29833984375 disk.drbd2165.in‚0 network.sent‚ 4.813017e+06# disk./.usageGB‚545.841495513916 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 160.66015625 proc.cpu.threads‚17 network.recv‚725870 disk.rbd1.out‚ 7.05859375# memory_percent‚95.4798936843872# proc.memory.rssMB‚ 2784.11328125) disk./.usagePercent‚36.63188075545899 disk.nvme0n1.in‚ 160.66015625" disk.nvme0c0n1.in‚ 160.66015625 disk.rbd3.out‚0# disk.dm-0.out‚11487.29833984375 cpu‚26.088534086179266 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 84.92578125 disk.dm-1.in‚0( proc.memory.availableMB‚ 740.57421875( disk.nvme0c0n1.out‚11487.29833984375 disk.sda.out‚ 84.92578125 disk.sda2.in‚0 disk.dm-1.out‚ 84.92578125* proc.memory.percent‚16.992878913879395 disk.rbd0.in‚0 disk.rbd0.out‚0(8»‘ž ‚(:† •Äçһʤ gpu.0.gpu‚51.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚278.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0® ³ŽŸ ‚(:Œ •ÄçÒšáÀ¦$ disk./.usageGB‚545.8427085876465 disk.sda.out‚ 86.92578125 disk.sda3.out‚ 86.92578125* proc.memory.percent‚16.992878913879395 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 7.12109375 cpu‚26.086466841772832 network.sent‚ 4.878197e+06 disk.rbd2.in‚0 disk.dm-1.out‚ 86.92578125 disk.rbd1.in‚0 disk.rbd3.in‚0$ memory_percent‚95.48001289367676' proc.memory.availableMB‚ 740.5546875) disk./.usagePercent‚36.63196216585487% disk.nvme0n1.out‚11503.4775390625 disk.sda1.in‚0 network.recv‚730278 disk.nvme0n1.in‚ 161.27734375 disk.sda1.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 161.27734375" disk.dm-0.out‚11503.4775390625# proc.memory.rssMB‚ 2784.11328125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 161.27734375' disk.nvme0c0n1.out‚11503.4775390625 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0ƪù ’  ‚(:‡ ¤ÄçÒ´“¥ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.775' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.155 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ¡¼(¡ ‚(:Œ ¤ÄçÒĺ…§ disk.sda.out‚ 87.38671875# disk.dm-0.out‚11514.05517578125* proc.memory.percent‚16.992878913879395 network.recv‚733510) disk./.usagePercent‚36.63186001884871 disk.sda2.in‚0 disk.sda3.out‚ 87.38671875 disk.rbd1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 7.12109375 disk.rbd0.in‚0 network.sent‚ 4.883274e+06$ disk./.usageGB‚545.8411865234375 disk.rbd2.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚11514.05517578125$ memory_percent‚96.98433876037598# proc.memory.rssMB‚ 2784.11328125 disk.sda.in‚0 disk.dm-1.out‚ 87.38671875 disk.nvme0n1.in‚ 161.9921875& disk.nvme0n1.out‚11514.05517578125 disk.drbd2165.out‚0 disk.rbd0.out‚0 cpu‚26.084333156384805 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 161.9921875 disk.sda2.out‚0' proc.memory.availableMB‚ 494.0859375 proc.cpu.threads‚17 disk.dm-0.in‚ 161.9921875 disk.drbd2165.in‚0Īü2¢ - _wandbruntime‚2693  _runtime‚2693E©ázÆ£ À & lr-AdamW/pg1‚0.0004500948872013424 & lr-AdamW/pg2‚0.0004500948872013424 trainer/global_step‚3149 $ _timestamp‚1.784275501730211e+09 _runtime‚2693.857498123 _step‚255Ó #ˤ Å & lr-AdamW/pg2‚0.0004500948872013424 trainer/global_step‚3149 $ _timestamp‚1.784275501730211e+09 _runtime‚2693.857498123 _step‚255 & lr-AdamW/pg1‚0.0004500948872013424ÿ?keæ2¥ - _wandbruntime‚2693  _runtime‚2693Á5u¦ ÚO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54008862 GADeviceStatsMonitor.on_train_batc„lûk&h_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105535952939008 _step‚256 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480670932258816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108890726532608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3313807942314 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚477352050876416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6754337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12274605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478217993064872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480670573880320 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3318540043264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29119540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5520268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29119540 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚477352033837056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3318881382400 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5520271 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474904525672448 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29129517 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480670573880320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3318881382400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108890840013312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842755017309666e+09ÏØyå'§ ßO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54008862 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105535952939008 _step‚256 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480670932258816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108890726532608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3313807942314 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚477352050876416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6754337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12274605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478217993064872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480670573880320 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3318540043264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29119540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5520268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29119540 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚477352033837056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3318881382400 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5520271 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474904525672448 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29129517 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚480670573880320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3318881382400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108890840013312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842755017309666e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144€¬ ç2¨ - _wandbruntime‚2694  _runtime‚2694™ž~ê&© äM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚477499291906048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 _step‚257 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480819185922560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478365514790466 _runtime‚2694.469969614 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚477499274866688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477499274866688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29128592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035539 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚475051053383680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3314818996148 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29138583 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480819185922560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29128592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5522016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚3149 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3319894016512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29138583 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3355807082496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025546 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24896954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478365872379828 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24896956 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6756401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12278692 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3314478446146 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24896956 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6756676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108925170167808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12278414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3319894016512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108925294127104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24896954 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 $ _timestamp‚1.784275502344196e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3319552670208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚477499291906048 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025546 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24896954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478365872379828 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/a«A~ctive_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24896956 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6756401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12278692 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3314478446146 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24896956 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6756676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108925170167808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12278414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3319894016512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108925294127104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24896954 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 $ _timestamp‚1.784275502344196e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3319552670208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚477499291906048 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035539 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚475051053383680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3314818996148ùÛ­2« - _wandbruntime‚2694  _runtime‚2694¶SËT¬ ü 2loss/SRqqt1/assignment_loss‚0.7071580868906913 /loss/FRt1/detection_loss‚0.5821114488867948 /loss/FRt2/detection_loss‚1.3547770880022447 /loss/FBt2/detection_loss‚1.1504672398524625 epoch‚7 $ _timestamp‚1.784275502345844e+09 /loss/FRt1/assignment_loss‚1.922748801136805 0loss/FRt2/assignment_loss‚0.9958733716904595 2loss/SRqqt2/assignment_loss‚0.5913025029641167 2loss/FBt1/assignment_loss‚0.023538470361149196 2loss/FBt2/assignment_loss‚0.030776480585408144 0loss/SRqqt2/detection_loss‚0.711784777921938 /loss/FBt1/detection_loss‚0.7229146476657617 trainer/global_step‚3149 0loss/SRqqt1/detection_loss‚1.023499619732373 %loss/total_loss‚9.816952535690204 _step‚258 _runtime‚ 2694.471172us—Ì­  $ _timestamp‚1.784275502345844e+09 /loss/FRt1/assignment_loss‚1.922748801136805 0loss/FRt2/assignment_loss‚0.9958733716904595 2loss/SRqqt2/assignment_loss‚0.5913025029641167 2loss/FBt1/assignment_loss‚0.023538470361149196 2loss/FBt2/assignment_loss‚0.030776480585408144 0loss/SRqqt2/detection_loss‚0.711784777921938 /loss/FBt1/detection_loss‚0.7229146476657617 trainer/global_step‚3149 0loss/SRqqt1/detection_loss‚1.023499619732373 %loss/total_loss‚9.816952535690204 _step‚258 _runtime‚ 2694.471172 2loss/SRqqt1/assignment_loss‚0.7071580868906913 /loss/FRt1/detection_loss‚0.5821114488867948 /loss/FRt2/detection_loss‚1.3547770880022447 /loss/FBt2/detection_loss‚1.1504672398524625 epoch‚7‚ܘ·“® ‚(:ˆ ³ÄçҌ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D:¯ ‚(: ³ÄçÒ™÷õ¤ disk.rbd0.in‚0 network.sent‚ 4.886196e+06) disk./.usagePercent‚36.63205842468775$ disk./.usageGB‚545.8441429138184 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚11521.29541015625 disk.sda3.out‚ 87.85546875 disk.dm-0.in‚ 162.8203125& disk.nvme0n1.out‚11521.29541015625 disk.rbd0.out‚0 cpu‚26.082366689532314 proc.cpu.threads‚17 disk.sda.out‚ 87.85546875 disk.rbd2.out‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 162.8203125( proc.memory.availableMB‚ 494.10546875 disk.rbd1.in‚0$ memory_percent‚96.98421955108643 network.recv‚737398 disk.sda.in‚0 disk.dm-1.out‚ 87.85546875 disk.sda1.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2784.11328125* proc.memory.percent‚16.992878913879395 disk.rbd1.out‚ 7.18359375 disk.rbd3.in‚0# disk.dm-0.out‚11521.29541015625 disk.rbd2.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 162.8203125ÿ˜‘° ‚(:’ ÂÄçÒœ¢Û¢ gpu.0.gpu‚58.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Ü¬Ë± ‚(:‘ ÂÄçÒÀ•Ê¦* disk./.usagePercent‚36.632102713991166& disk.nvme0n1.out‚11530.93115234375" disk.nvme0c0n1.in‚ 163.53515625 disk.sda3.in‚0 disk.rbd3.in‚0 network.recv‚741800 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 7.18359375 disk.nvme0n1.in‚ 163.53515625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 493.66796875 disk.sda1.in‚0$ disk./.usageGB‚545.8448028564453 disk.sda3.out‚ 88.06640625 disk.dm-1.in‚0 disk.dm-0.in‚ 163.53515625# disk.dm-0.out‚11530.93115234375* proc.memory.percent‚16.992878913879395 cpu‚26.080346940502984 disk.rbd2.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚11530.93115234375 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 4.951292e+06 disk.rbd2.out‚0 disk.sda.out‚ 88.06640625 disk.dm-1.out‚ 88.06640625 disk.rbd3.out‚0$ memory_percent‚96.98688983917236# proc.memory.rssMB‚ 2784.11328125 proc.cpu.threads‚17Iin`›² ‚(: ÑÄçÒàÌœ¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.224' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.84479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04Nû³ ‚(:ð ÑÄçÒ嶘¦ network.recv‚744535 disk.nvme0n1.in‚ 163.53515625 network.sent‚ 4.95543e+06 disk.rbd0.in‚0 disk.rbd2.in‚0 cpu‚26.078479118580972# disk./.usageGB‚545.844783782959 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.out‚88.40625 disk.dm-0.out‚ 11535.90625 disk.rbd2.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.632101433953494 disk.dm-0.in‚ 163.53515625# memory_percent‚96.9839096069336 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd3.out‚0* proc.memory.percent‚16.992878913879395" disk.nvme0c0n1.in‚ 163.53515625" disk.nvme0c0n1.out‚ 11535.90625 disk.sda3.in‚0 disk.sda3.out‚88.40625 disk.rbd1.out‚ 7.24609375 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2784.11328125 disk.sda2.in‚0 disk.dm-1.out‚88.40625 disk.nvme0n1.out‚ 11535.90625 disk.sda1.in‚0% proc.memory.availableMB‚ 494.15625'v±Ü2´ - _wandbruntime‚2733  _runtime‚2733•±’úǵ Á % _timestamp‚1.7842755416387453e+09 _runtime‚2733.764331724 _step‚259 & lr-AdamW/pg1‚0.0004492802129289349 & lr-AdamW/pg2‚0.0004492802129289349 trainer/global_step‚3199¶d̶ Æ & lr-AdamW/pg1‚0.0004492802129289349 & lr-AdamW/pg2‚0.0004492802129289349 trainer/global_step‚3199 % _timestamp‚1.7842755416387453e+09 _runtime‚2733.764331724 _step‚259ƒS(–2· - _wandbruntime‚2733  _runtime‚2733êK56à'¸ ÚO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3405986889216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3364424429222 trainer/global_step‚3199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484714102358016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485594500734630 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482230059266048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3369576884224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488083679242240 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25270924 _runtime‚2733.765585981 _step‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 % _timestamp‚1.7842755416397932e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25270922 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3369235545088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484714085318656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25270922 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6858060 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110613272662528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107207372253184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29573992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484714085318656 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110613377754624 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29583969 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12465841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107207390865408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12465578 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488083320863744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484714102358016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29583969 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154½¨–ž"¹ ßO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6858320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485594143145380 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5607518 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3405900409344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5607521 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚488083320863744 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3369235545088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25270924 _runtime‚2733.765585981 _step‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 % _timestamp‚1.7842755416397932e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25270922 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3369235545088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484714085318656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25270922 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6858060 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110613272662528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107207372253184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29573992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484714085318656 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110613377754624 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29583969 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12465841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107207390865408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12465578 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488083320863744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484714102358016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29583969 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3405986889216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3364424429222 trainer/global_step‚3199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484714102358016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485594500734630 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482230059266048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3369576884224 QIDeviceStatsMonitor.on_train_batch_start/inÖyßactive_split.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488083679242240 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6860390 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488231574520320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107240801011712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107240819623936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25278554 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5609263 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482376586977280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482376569937920 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861598 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3365435483056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488231932905984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370589518336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110647831895040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370248172032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 _step‚261 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484861343387648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29583044 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3406913215488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2734.379687541 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25278556 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 trainer/global_step‚3199 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12469389 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370589518336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29583044 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484861326348288 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29593035 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488231932905984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3407012271104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485742022460336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484861343387648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269æÁùŽï&¼ éM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29583044 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484861326348288 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29593035 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488231932905984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3407012271104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚485742022460336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484861343387648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12469656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110647714227200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6860390 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488231574520320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107240801011712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107240819623936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25278554 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5609263 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482376586977280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482376569937920 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861598 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3365435483056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488231932905984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370589518336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110647831895040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370248172032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 _step‚261 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484861343387648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29583044 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3406913215488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2734.379687541 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25278556 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 trainer/global_step‚3199 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12469389 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370589518336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144….2½ - _wandbruntime‚2734  _runtime‚2734…â ¾ ƒ 1loss/SRqqt1/detection_loss‚1.0107558695850016 _step‚262 2loss/SRqqt1/assignment_loss‚0.6031302996999073 2loss/SRqqt2/assignment_loss‚0.6100235025168853 2loss/FBt2/assignment_loss‚0.008606298034780389 1loss/SRqqt2/detection_loss‚0.6712475143096835 /loss/FBt1/detection_loss‚0.5882161175565779 /loss/FBt2/detection_loss‚0.6460273477338976 %loss/total_loss‚8.893100688704642 % _timestamp‚1.7842755422559652e+09 0loss/FRt1/assignment_loss‚1.9407894400135095 0loss/FRt2/assignment_loss‚1.1258032820296526 2loss/FBt1/assignment_loss‚0.008537873230914962 /loss/FRt1/detection_loss‚0.5226825510938278 /loss/FRt2/detection_loss‚1.1572805929000032 trainer/global_step‚3199 _runtime‚2734.381323248 epoch‚7¢?g¿ ˆ 1loss/SRqqt1/detection_loss‚1.0107558695850016 _step‚262 2loss/SRqqt1/assignment_loss‚0.6031302996999073 2loss/SRqqt2/assignment_loss‚0.6100235025168853 2loss/FBt2/assignment_loss‚0.008606298034780389 1loss/SRqqt2/detection_loss‚0.6712475143096835 /loss/FBt1/detection_loss‚0.5882161175565779 /loss/FBt2/detection_loss‚0.6460273477338976 %loss/total_loss‚8.893100688704642 % _timestamp‚1.7842755422559652e+09 0loss/FRt1/assignment_loss‚1.9407894400135095 0loss/FRt2/assignment_loss‚1.1258032820296526 2loss/FBt1/assignment_loss‚0.008537873230914962 /loss/FRt1/detection_loss‚0.5226825510938278 /loss/FRt2/detection_loss‚1.1572805929000032 trainer/global_step‚3199 _runtime‚2734.381323248 epoch‚7†4tR“À ‚(:ˆ àÄçÒ‹…Ù¥ gpu.0.gpu‚96.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦üÁ ‚(:Ž àÄçÒõ˜¨ disk.dm-1.in‚0 network.sent‚ 4.971388e+06 network.recv‚747916$ disk./.usageGB‚545.8448486328125 disk.sda.out‚ 88.60546875 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚26.092174273128517 disk.sda2.out‚0 disk.dm-1.out‚ 88.60546875& proc.memory.availableMB‚ 493.671875 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚36.63210578608158 disk.sda1.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.992878913879395 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚11546.11669921875 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 7.24609375 disk.dm-0.in‚ 164.36328125& disk.nvme0n1.out‚11546.11669921875" disk.nvme0c0n1.in‚ 164.36328125 disk.rbd2.in‚0# proc.memory.rssMB‚ 2784.11328125 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚11546.11669921875 disk.nvme0n1.in‚ 164.36328125 proc.cpu.threads‚17 disk.sda3.out‚ 88.60546875 disk.rbd2.out‚0$ memory_percent‚96.98686599731445Þ4ù°2 - _wandbruntime‚2751  _runtime‚2751 ¢à ­{ particle/accuracy_2_of_2‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960519671440125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8020598292350769 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7506445646286011 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 particle/accuracy_1_of_2‚1 particle/accuracy_5_of_5‚0 $ _timestamp‚1.784275559346138e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.799561619758606 particle/accuracy_1_of_1‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8209547996520996 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2776053249835968 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.653610110282898 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.904998779296875 particle/accuracy_1_of_4‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21573375165462494 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9939030408859253 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.973479151725769 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6388538479804993 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3953738808631897 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2203669548034668 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22680261731147766 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9916179180145264 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.33228766918182373 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933305382728577 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19479534029960632 +jet/accuracy_2_of_2‚0.14942839741706848 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19667933881282806 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04559706896543503 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5429166555404663 *jet/accuracy_3_of_3‚0.0726630762219429 *jet/accuracy_2_of_4‚0.7761441469192505 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11245454847812653 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3055555820465088 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16452108323574066 particle/accuracy_1_of_3‚1 (loss/val_total_loss‚9.10819503401277 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7840136289596558 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04703310504555702 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7223191261291504 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3055555820465088 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6846408843994141 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7666124701499939 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.044140107929706573 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19196249544620514 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6654465198516846 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6099357604980469 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 _runtime‚2751.472474861 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9066706895828247 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3983941674232483 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0949329361319542 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.712397575378418 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9877275824546814 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7480431199073792 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9923742413520813 'particle/f_score‚0.6302911639213562 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5671133399009705 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9843167662620544 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1932959109544754 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18022814393043518 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9902719855308533 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43317845463752747 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16083332896232605 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9805901646614075 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7145295143127441 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_2_of_3‚0.5739374160766602 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7888889312744141 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47367435693740845 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6466718912124634 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6586347222328186 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7698602080345154 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6704455018043518 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04037903621792793 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8209547996520996 +particle/sensitivity‚0.6112778782844543 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7117348909378052 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7557215690612793 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32168084383010864 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 (particle/accuracy‚0.8460091352462769 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6767642498016357 particle/accuracy_1_of_5‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960519671440125 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.801157534122467 =&Purity/*FRt0SRqq-¬pt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7026832699775696 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7507990598678589 *jet/accuracy_1_of_3‚0.9268091320991516 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8704416155815125 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7612512707710266 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7638024687767029 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22802066802978516 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7155865430831909 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6886453628540039 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9611803889274597 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6407172083854675 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4455729126930237 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23745189607143402 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21387486159801483 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036414794623851776 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7843083143234253 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.764621913433075 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1778840273618698 +particle/specificity‚0.9102049469947815 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.045854274183511734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13052044808864594 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 6validation_average_jet_accuracy‚0.3743111193180084 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4871068596839905 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.967302680015564 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9711123704910278 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9904478192329407 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8059713244438171 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04852431267499924 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04703310504555702 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9802785515785217 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48844438791275024 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5932145118713379 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04450384899973869 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21182753145694733 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7127809524536133 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17527882754802704 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6614583134651184 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957615733146667 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7213993072509766 .particle/accuracy_4_of_4‚0.938076376914978 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.193142369389534 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6676949858665466 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22719618678092957 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025036420673131943 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2850162982940674 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21573375165462494 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11279019713401794 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4880683422088623 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3854166865348816 trainer/global_step‚3199 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6704455018043518 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32406073808670044 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6689754724502563 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.021724846214056015 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8479877710342407 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5736027956008911 *jet/accuracy_1_of_2‚0.7916582226753235 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1798991858959198 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19317232072353363 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013692273758351803 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7866335511207581 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6704452037811279 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9898972511291504 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4615384638309479 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472222089767456 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4615384638309479 _step‚263 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20755888521671295 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7061518430709839 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870816469192505 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6646787524223328 +jet/accuracy_3_of_4‚0.08173758536577225 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7692307829856873 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 epoch‚7 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8942397236824036 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 *jet/accuracy_1_of_4‚0.9690386652946472ô35E¸=Ä ²{ 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6767642498016357 particle/accuracy_1_of_5‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960519671440125 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.801157534122467 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7026832699775696 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7507990598678589 *jet/accuracy_1_of_3‚0.9268091320991516 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8704416155815125 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7612512707710266 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7638024687767029 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22802066802978516 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7155865430831909 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6886453628540039 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9611803889274597 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6407172083854675 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4455729126930237 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23745189607143402 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21387486159801483 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036414794623851776 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7843083143234253 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.764621913433075 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1778840273618698 +particle/specificity‚0.9102049469947815 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.045854274183511734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13052044808864594 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 6validation_average_jet_accuracy‚0.3743111193180084 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4871068596839905 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.967302680015564 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9711123704910278 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9904478192329407 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8059713244438171 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04852431267499924 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04703310504555702 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9802785515785217 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48844438791275024 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5932145118713379 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04450384899973869 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21182753145694733 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7127809524536133 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17527882754802704 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6614583134651184 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957615733146667 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7213993072509766 .particle/accuracy_4_of_4‚0.938076376914978 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.193142369389534 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6676949858665466 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22719618678092957 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025036420673131943 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2850162982940674 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21573375165462494 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11279019713401794 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4880683422088623 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3854166865348816 trainer/global_step‚3199 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6704455018043518 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32406073808670044 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6689754724502563 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.021724846214056015 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8479877710342407 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5736027956008911 *jet/accuracy_1_of_2‚0.7916582226753235 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1798991858959198 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19317232072353363 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013692273758351803 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7866335511207581 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6704452037811279 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9898972511291504 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4615384638309479 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472222089767456 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4615384638309479 _step‚263 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20755888521671295 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7061518430709839 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870816469192505 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6646787524223328 +jet/accuracy_3_of_4‚0.08173758536577225 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7692307829856873 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 epoch‚7 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8942397236824036 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 *jet/accuracy_1_of_4‚0.9690386652946472 particle/accuracy_2_of_2‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960519671440125 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8020598292350769 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7506445646286011 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 particle/accuracy_1_of_2‚1 particle/accuracy_5_of_5‚0 $ _timestamp‚1.784275559346138e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.799561619758606 particle/accuracy_1_of_1‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8209547996520996 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2776053249835968 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.653610110282898 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.904998779296875 particle/accuracy_1_of_4‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21573375165462494 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9939030408859253 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.973479151725769 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6388538479804993 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3953738808631897 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2203669548034668 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22680261731147766 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9916179180145264 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.33228766918182373 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9933305382728577 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19479534029960632 +jet/accuracy_2_of_2‚0.14942839741706848 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19667933881282806 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04559706896543503 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5429166555404663 *jet/accuracy_3_of_3‚0.0726630762219429 *jet/accuracy_2_of_4‚0.7761441469192505 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11245454847812653 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3055555820465088 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16452108323574066 particle/accuracy_1_of_3‚1 (loss/val_total_loss‚9.10819503401277 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7840136289596558 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04703310504555702 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7223191261291504 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3055555820465088 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6846408843994141 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7666124701499939 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.044140107929706573 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19196249544620514 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6654465198516846 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6099357604980469 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 _runtime‚2751.472474861 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9066706895828247 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3983941674232483 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0949329361319542 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.712397575378418 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9877275824546814 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7480431199073792 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9923742413520813 'particle/f_score‚0.6302911639213562 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5671133399009705 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9843167662620544 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1932959109544754 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18022814393043518 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9902719855308533 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43317845463752747 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16083332896232605 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9805901646614075 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7145295143127441 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_2_of_3‚0.5739374160766602 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7888889312744141 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47367435693740845 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6466718912124634 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6586347222328186 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7698602080345154 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6704455018043518 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04037903621792793 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8209547996520996 +particle/sensitivity‚0.6112778782844543 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7117348909378052 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7557215690612793 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32168084383010864 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 (particle/accuracy‚0.8460091352462769 jet/accuracy_3_of_5‚1‡$º« žÅ ‚(:“ ïÄçÒÿÆ·£ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏL|Æ ‚(:ù ïÄçÒóÖŒ¦ disk.sda.out‚ 88.87890625 disk.dm-0.in‚ 165.078125# disk.nvme0n1.out‚11799.11328125 cpu‚26.09867145892185 disk.sda3.in‚0% proc.memory.availableMB‚ 743.15625* disk./.usagePercent‚36.632191036590484 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0! proc.memory.rssMB‚ 2787.234375 network.recv‚752544 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-0.out‚11799.11328125 disk.sda1.out‚0# memory_percent‚95.4641342163086 disk.sda3.out‚ 88.87890625 disk.dm-1.out‚ 88.87890625 disk.sda1.in‚0 disk.drbd2165.in‚0 network.sent‚ 5.034167e+06# disk./.usageGB‚545.846118927002 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 165.078125 disk.sda2.in‚0 disk.rbd2.out‚0 disk.rbd1.out‚ 7.24609375 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 165.078125 disk.drbd2165.out‚0) proc.memory.percent‚17.01192855834961 proc.cpu.threads‚17% disk.nvme0c0n1.out‚11799.11328125?̵]“Ç ‚(:ˆ þÄçÒžõѤ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚238.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.6244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^—ò}È ‚(:ƒ þÄçÒسÿ¥( disk./.usagePercent‚36.6320909376446( disk.nvme0c0n1.out‚11873.03564453125 disk.sda3.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚756369 disk.sda.out‚ 89.01953125$ memory_percent‚95.46115398406982 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 7.30859375 disk.dm-0.in‚ 165.90625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 165.90625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0# disk.dm-0.out‚11873.03564453125 disk.sda1.in‚0) proc.memory.percent‚17.01192855834961 network.sent‚ 5.078678e+06$ disk./.usageGB‚545.8446273803711 disk.rbd0.out‚0 disk.dm-1.out‚ 89.01953125 disk.rbd3.out‚0 disk.rbd2.out‚0& disk.nvme0n1.out‚11873.03564453125 cpu‚26.096641789377816 disk.sda3.out‚ 89.01953125 disk.dm-1.in‚0! proc.memory.rssMB‚ 2787.234375 disk.nvme0c0n1.in‚ 165.90625 disk.rbd3.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 743.64453125µRcV’É ‚(:‡ Åçҥ⛥ gpu.0.gpu‚10.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%K,µ Ê ‚(: ÅçÒŨƧ# disk.dm-0.out‚12058.08837890625 disk.nvme0n1.in‚ 165.90625 network.sent‚ 5.083749e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd2.out‚0& disk.nvme0n1.out‚12058.08837890625$ memory_percent‚95.46115398406982( proc.memory.availableMB‚ 743.64453125 cpu‚26.094601799294253 disk.rbd0.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚759843) disk./.usagePercent‚36.63208632950898 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2787.234375$ disk./.usageGB‚545.8445587158203 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12058.08837890625 disk.sda3.out‚ 89.2890625 disk.dm-1.out‚ 89.2890625 disk.rbd1.out‚ 7.30859375 disk.sda1.in‚0) proc.memory.percent‚17.01192855834961 disk.nvme0c0n1.in‚ 165.90625 disk.sda.in‚0 disk.sda.out‚ 89.2890625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 165.90625 disk.rbd2.in‚0b(X32Ë - _wandbruntime‚2792  _runtime‚2792Z[´ÀÇÌ Á & lr-AdamW/pg1‚0.0004484518615260634 & lr-AdamW/pg2‚0.0004484518615260634 trainer/global_step‚3249 % _timestamp‚1.7842756007180264e+09 _runtime‚2792.843608693 _step‚264FCP ÌÍ Æ trainer/global_step‚3249 % _timestamp‚1.7842756007180264e+09 _runtime‚2792.843608693 _step‚264 & lr-AdamW/pg1‚0.0004484518615260634 & lr-AdamW/pg2‚0.0004484518615260634ˆþ±óì2Î - _wandbruntime‚2792  _runtime‚2792&‹ucÏ ÙO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30108748 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497443726084096 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3475391990784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30108748 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112370114270208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12690625 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494005720489984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30098771 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108894740891648 % _timestamp‚1.7842756007190747e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25773188 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3437664254976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25773190 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5697288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6993334 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112370208876544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494005703450624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3432420091474 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55871959 A7DeviceStatsMonitor.on_train_batch_staš6ðmÛrt/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚265 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30098771 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3432760641364 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497443726084096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚494005720489984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3475467984896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25773190 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108894722279424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3438005594112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497443367705600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12690359 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3437664254976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55881938 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5697291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494896676165202 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6993071 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497443367705600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚494005703450624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚491464273113088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25773188 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚491464256073728 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ©^Îä'Ð ÞO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3438005594112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494897033754452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3437664254976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55881938 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5697291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494896676165202 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6993071 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497443367705600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚494005703450624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚491464273113088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25773188 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚491464256073728 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30108748 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497443726084096 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3475391990784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30108748 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112370114270208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12690625 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494005720489984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30098771 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108894740891648 % _timestamp‚1.7842756007190747e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25773188 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3437664254976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25773190 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5697288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6993334 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112370208876544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494005703450624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3432420091474 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55871959 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚265 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30098771 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3432760641364 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497443726084096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚494005720489984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3475467984896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25773190 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108894722279424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3438005594112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497443367705600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12690359 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚495044197890796 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898636 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 # _timestamp‚1.78427560133286e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3438676881920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3476420930560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3439018228224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3438676881920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3433771695198 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30107823 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25780822 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25780820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6995056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30117814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112404571968512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25780822 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3476503208960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12694352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3439018228224 _runtime‚2793.458752111 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚494152961519616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30107823 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚3249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5699033 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12694089 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112404672859136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495044555480158 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30117814 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497591621362176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497591979747840 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6995316 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚3249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5699033 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12694089 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112404672859136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495044555480158 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30117814 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497591621362176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497591979747840 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6995316 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚495044197890796 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898636 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsM£ié onitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 # _timestamp‚1.78427560133286e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3438676881920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3476420930560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3439018228224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3438676881920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3433771695198 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30107823 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25780822 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25780820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6995056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30117814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112404571968512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25780822 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3476503208960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Š*S 2Ô - _wandbruntime‚2793  _runtime‚2793A¢aŽÕ € _step‚267 2loss/FBt2/assignment_loss‚0.002797727020362196 $ _timestamp‚1.784275601335324e+09 2loss/FBt1/assignment_loss‚0.035236979282946175 1loss/SRqqt1/detection_loss‚1.0904400895975916 1loss/SRqqt2/detection_loss‚0.6763278814969854 epoch‚8 /loss/FRt1/assignment_loss‚1.939865895165693 /loss/FRt2/detection_loss‚1.2526306829129188 /loss/FBt1/detection_loss‚0.5910825144867373 trainer/global_step‚3249 /loss/FRt2/assignment_loss‚1.079369604587555 2loss/SRqqt1/assignment_loss‚0.6578936043979724 2loss/SRqqt2/assignment_loss‚0.6239137342957939 /loss/FRt1/detection_loss‚0.5558264223828829 /loss/FBt2/detection_loss‚0.7298281237276064 %loss/total_loss‚9.235213259355046 _runtime‚2793.460649693ãÃX´ Ö … /loss/FRt2/assignment_loss‚1.079369604587555 2loss/SRqqt1/assignment_loss‚0.6578936043979724 2loss/SRqqt2/assignment_loss‚0.6239137342957939 /loss/FRt1/detection_loss‚0.5558264223828829 /loss/FBt2/detection_loss‚0.7298281237276064 %loss/total_loss‚9.235213259355046 _runtime‚2793.460649693 _step‚267 2loss/FBt2/assignment_loss‚0.002797727020362196 $ _timestamp‚1.784275601335324e+09 2loss/FBt1/assignment_loss‚0.035236979282946175 1loss/SRqqt1/detection_loss‚1.0904400895975916 1loss/SRqqt2/detection_loss‚0.6763278814969854 epoch‚8 /loss/FRt1/assignment_loss‚1.939865895165693 /loss/FRt2/detection_loss‚1.2526306829129188 /loss/FBt1/detection_loss‚0.5910825144867373 trainer/global_step‚3249‹âùÜâ“× ‚(:ˆ œÅçÒ§Âî¥ gpu.0.gpu‚35.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú *ÂØ ‚(:† œÅçÒÊ­À¨ disk.sda.in‚0 disk.sda.out‚ 89.6953125 disk.sda2.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0) proc.memory.percent‚17.01192855834961 disk.sda2.out‚0 disk.rbd1.out‚ 7.43359375 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0! proc.memory.rssMB‚ 2787.234375 network.sent‚ 5.088878e+06 network.recv‚762989 disk.rbd3.out‚0 disk.dm-0.in‚ 166.62109375" disk.dm-0.out‚12068.0888671875 disk.rbd2.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 743.6484375) disk./.usagePercent‚36.63194578137268 disk.sda3.out‚ 89.6953125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 166.62109375' disk.nvme0c0n1.out‚12068.0888671875 disk.rbd1.in‚0$ memory_percent‚95.46113014221191 cpu‚26.092682959259946$ disk./.usageGB‚545.8424644470215 disk.sda3.in‚0% disk.nvme0n1.out‚12068.0888671875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 166.62109375 disk.dm-1.out‚ 89.6953125 proc.cpu.threads‚176j(hžÙ ‚(:“ «Åçҟᯥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ãÚ ‚(:ˆ «ÅçÒ¯¢« network.recv‚767394 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 90.01171875 disk.sda2.in‚0 disk.dm-0.in‚ 167.44921875 disk.nvme0n1.in‚ 167.44921875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 90.01171875 disk.dm-1.in‚0! disk.dm-0.out‚12075.185546875 cpu‚26.090837064009133 disk.sda1.out‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚12075.185546875 disk.dm-1.out‚ 90.01171875 disk.rbd1.in‚0( proc.memory.availableMB‚ 743.71484375* disk./.usagePercent‚36.631940661221996" disk.nvme0c0n1.in‚ 167.44921875$ memory_percent‚95.46072483062744$ disk./.usageGB‚545.8423881530762 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.01192855834961 proc.cpu.threads‚17 network.sent‚ 5.153951e+06 disk.rbd2.in‚0 disk.rbd1.out‚ 7.43359375! proc.memory.rssMB‚ 2787.234375 disk.rbd2.out‚0$ disk.nvme0n1.out‚12075.185546875 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda3.in‚0ãKøG2Û - _wandbruntime‚2832  _runtime‚2832"ê·6ÈÜ Â _step‚268 ' lr-AdamW/pg1‚0.00044760988619619083 ' lr-AdamW/pg2‚0.00044760988619619083 trainer/global_step‚3299 $ _timestamp‚1.784275640651565e+09 _runtime‚2832.778426701ƒCTÍÝ Ç ' lr-AdamW/pg1‚0.00044760988619619083 ' lr-AdamW/pg2‚0.00044760988619619083 trainer/global_step‚3299 $ _timestamp‚1.784275640651565e+09 _runtime‚2832.778426701 _step‚268ŒÒS˜2Þ - _wandbruntime‚2832  _runtime‚2832 ú3¯à'ß ÚO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504856114689024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3488701095936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30563200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚3299 % _timestamp‚1.7842756406523347e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717990 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7095903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708011 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110566178818048 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3483036578382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501367771971584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30553223 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5784541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26154788 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3526843527680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501367771971584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114092919350784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502272826245710 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976ëðå'à ßO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110566178818048 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3483036578382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501367771971584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30553223 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5784541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26154788 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3526843527680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501367771971584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114092919350784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502272826245710 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504856114689024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3488701095936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30563200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚3299 % _timestamp‚1.7842756406523347e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717990 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7095903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708011 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438ŒC2á - _wandbruntime‚2833  _runtime‚2833õ®â äM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12884205 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚2833.390864507 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501514995961856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842756412649906e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501515013001216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26162420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3527779930112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7097922 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMo{«Ënitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505004726731264 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734688 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114127478173696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3299 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30562275 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114127368894464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚270 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3484388182106 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498936317378560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚502420347971304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30572266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5786286 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚05/,³ï&ã éM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3489372383744 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12884471 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3489713730048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724695 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3527870597120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26162422 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501514995961856 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚270 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3484388182106 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚498936317378560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚502420347971304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30572266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5786286 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12884205 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚2833.390864507 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501514995961856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842756412649906e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501515013001216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26162420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3527779930112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7097922 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505004726731264 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734688 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114127478173696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3299 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30562275 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114127368894464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512268861672448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539396597760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3539055258624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3533993615180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57544063 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31007675 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115815776116736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115815662636032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2872.706519507 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31007675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508729806413824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3539055258624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512269220050944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57554042 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚508729823453184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26536388 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112237616744448 trainer/global_step‚3349 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5871788 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506115323260928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842756805802119e+09 _step‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26536390 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539396597760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚512269220050944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7199028 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26536388 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31017652 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506115340300288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13071069 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:Device ¡¸;StatsMonitor.on_train_batch_start/active.all.current‚9979*Ûu/å'ò ßO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚508729823453184 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3578064503808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3533653065290 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512268861672448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539396597760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3539055258624 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3533993615180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57544063 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31007675 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115815776116736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115815662636032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2872.706519507 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31007675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508729806413824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3539055258624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512269220050944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57554042 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚508729823453184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26536388 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112237616744448 trainer/global_step‚3349 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5871788 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506115323260928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842756805802119e+09 _step‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26536390 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539396597760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚512269220050944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7199028 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26536388 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31017652 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506115340300288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13071069 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560747 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚274 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5873536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579084781056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚506261850972160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚512417473714688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579179642368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3535004669014 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13074538 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31016727 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112271026890752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚509796855641174 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚508877064482816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3540067885568 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31026718 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540409231872 trainer/global_step‚3349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26544022 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5873533 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115850111671808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚512417115329024 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚508877047443456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3540067885568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570740 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7201005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚512417473714688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26544020 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31016727 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13074822 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26544022 $ _timestamp‚1.784275681193451e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26544020 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560747 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚508877047443456 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13074538 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31016727 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112271026890752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚509796855641174 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚508877064482816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3540067885568 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31026718 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540409231872 trainer/global_step‚3349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26544022 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5873533 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115850111671808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚512417115329024 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚508877047443456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3540067885568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570740 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7201005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚512417473714688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26544020 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31016727 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13074822 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26544022 $ _timestamp‚1.784275681193451e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26544020 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560747 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚508877047443456 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560747 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚274 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5873536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579084781056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚506261850972160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚512417473714688 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579179642368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3535004669014’µËO·2ö - _wandbruntime‚2873  _runtime‚2873ÍÑÂi÷ € epoch‚8 %loss/total_loss‚9.065368332646246 2loss/SRqqt1/assignment_loss‚0.5713394073427024 2loss/SRqqt2/assignment_loss‚0.5727689282646038 1loss/SRqqt2/detection_loss‚0.6529283381983517 .loss/FBt1/detection_loss‚0.498694538758675 /loss/FBt2/detection_loss‚0.9301823228597641 _runtime‚2873.320953895 _step‚275 /loss/FRt1/detection_loss‚0.5209701512847632 0loss/FRt1/assignment_loss‚1.9496132866961176 1loss/FBt2/assignment_loss‚0.06276104134542493 /loss/FRt2/detection_loss‚1.2520884650709656 trainer/global_step‚3349 % _timestamp‚1.7842756811956236e+09 0loss/FRt2/assignment_loss‚0.9501060412216872 2loss/FBt1/assignment_loss‚0.027620845288376748 0loss/SRqqt1/detection_loss‚1.076294966314814#ëV ø … 2loss/SRqqt2/assignment_loss‚0.5727689282646038 1loss/SRqqt2/detection_loss‚0.6529283381983517 .loss/FBt1/detection_loss‚0.498694538758675 /loss/FBt2/detection_loss‚0.9301823228597641 _runtime‚2873.320953895 _step‚275 /loss/FRt1/detection_loss‚0.5209701512847632 0loss/FRt1/assignment_loss‚1.9496132866961176 1loss/FBt2/assignment_loss‚0.06276104134542493 /loss/FRt2/detection_loss‚1.2520884650709656 trainer/global_step‚3349 % _timestamp‚1.7842756811956236e+09 0loss/FRt2/assignment_loss‚0.9501060412216872 2loss/FBt1/assignment_loss‚0.027620845288376748 0loss/SRqqt1/detection_loss‚1.076294966314814 epoch‚8 %loss/total_loss‚9.065368332646246 2loss/SRqqt1/assignment_loss‚0.5713394073427024“/´"užù ‚(:“ çÅçÒ¿š¾§ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BcRß°ú ‚(:„ çÅçÒ™€Ùª disk.sda2.in‚0" disk.nvme0c0n1.in‚ 169.70703125 network.recv‚782055) disk./.usagePercent‚36.63198981466857 disk.rbd3.out‚0$ memory_percent‚95.46091556549072 disk.sda3.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚12109.9482421875! proc.memory.rssMB‚ 2787.234375 disk.nvme0n1.in‚ 169.70703125 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 7.55859375 d”c‘G_isk.rbd0.out‚0( proc.memory.availableMB‚ 743.68359375% disk.nvme0n1.out‚12109.9482421875 disk.sda1.out‚0 disk.drbd2165.in‚0 network.sent‚ 5.229932e+06$ disk./.usageGB‚545.8431205749512" disk.dm-0.out‚12109.9482421875 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.dm-1.out‚ 92.703125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 92.703125 cpu‚26.083087411959994 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 92.703125 disk.rbd3.in‚0 disk.dm-0.in‚ 169.70703125 disk.sda1.in‚0) proc.memory.percent‚17.01192855834961)ÞÇžû ‚(:“ öÅçÒÀµÓ¢ gpu.0.gpu‚62.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i<´­ ü ‚(: öÅçÒÈÀš§ disk.dm-0.out‚ 12122.0546875* disk./.usagePercent‚36.631883571541856 disk.nvme0n1.in‚ 170.53515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.01192855834961 network.sent‚ 5.295129e+06 disk.dm-0.in‚ 170.53515625" disk.nvme0n1.out‚ 12122.0546875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 170.53515625$ memory_percent‚95.46055793762207' proc.memory.availableMB‚ 743.7421875$ disk./.usageGB‚545.8415374755859 disk.rbd2.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 93.57421875 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚786612 disk.rbd2.in‚0 disk.sda.out‚ 93.57421875 disk.sda3.in‚0 disk.sda3.out‚ 93.57421875 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 12122.0546875 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 7.62109375! proc.memory.rssMB‚ 2787.234375 cpu‚26.081197991227796C1Üe‘ý ‚(:† …ÆçÒÏ·ù¤ gpu.0.gpu‚71.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.47' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.294 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!0Êþþ ‚(:ó …ÆçÒ½ßã¦( proc.memory.availableMB‚ 743.73828125 proc.cpu.threads‚17$ disk./.usageGB‚545.8414268493652 disk.sda2.out‚0! proc.memory.rssMB‚ 2787.234375) proc.memory.percent‚17.01192855834961 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 93.8828125 disk.rbd1.in‚0 disk.dm-0.in‚171.25 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 7.62109375 disk.rbd2.in‚0 disk.nvme0c0n1.in‚171.25) disk./.usagePercent‚36.63187614732337 disk.sda1.in‚0 network.recv‚789855 disk.rbd3.in‚0" disk.dm-0.out‚12129.0439453125 disk.rbd2.out‚0 disk.nvme0n1.in‚171.25' disk.nvme0c0n1.out‚12129.0439453125 cpu‚26.0792711673333 network.sent‚ 5.300167e+06 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 93.8828125 disk.dm-1.out‚ 93.8828125% disk.nvme0n1.out‚12129.0439453125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚95.46058177947998ß.=2ÿ - _wandbruntime‚2912  _runtime‚2912¨¤VhÇ€Á % _timestamp‚1.7842757203276422e+09 _runtime‚2912.453317945 _step‚276 & lr-AdamW/pg1‚0.0004458852809410162 & lr-AdamW/pg2‚0.0004458852809410162 trainer/global_step‚3399<ÖÞÌÆ % _timestamp‚1.7842757203276422e+09 _runtime‚2912.453317945 _step‚276 & lr-AdamW/pg1‚0.0004458852809410162 & lr-AdamW/pg2‚0.0004458852809410162 trainer/global_step‚3399”Èö^x2‚- _wandbruntime‚2912  _runtime‚2912å .)à'ƒÚO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513440856854528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519681967034368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚516091857895424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26917990 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7302499 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519681608655872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517025126406726 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390094 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26917988 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3629474590720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113909036058624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519681608655872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚513440873893888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3629390208000 _runtime‚2912.454536546 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31472104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3590092099584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26917988 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3629474590720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113909036058624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519681608655872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚513440873893888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3629390208000 _runtime‚2912.454536546 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31472104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3590092099584 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26917990 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7302499 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519681608655872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517025126406726 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390094 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977•ãDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13265294 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 % _timestamp‚1.7842757209421601e+09 _runtime‚ 2913.06795487 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113942464817152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31471179 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31471179 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26925622 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516239115964416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117572977487360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26925620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3585621155922 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3591104733696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516239098925056 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7304778 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517172648132320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26925622 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3630399196672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406792 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7304511 XJDeviceStatsMonitor.on_train_batch_end/©eÊinactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚3399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513587367526400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3590763387392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516239098925056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117572864013824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31481170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113942483429376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517173005721682 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 0Šî&‡èM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513587367526400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3590763387392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516239098925056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117572864013824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31481170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113942483429376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517173005721682 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26925620 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3590763387392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13265294 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 % _timestamp‚1.7842757209421601e+09 _runtime‚ 2913.06795487 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113942464817152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31471179 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31471179 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26925622 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516239115964416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117572977487360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26925620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3585621155922 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3591104733696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516239098925056 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7304778 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517172648132320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26925622 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3630399196672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406792 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7304511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚3399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0–aÈ2ˆ- _wandbruntime‚2913  _runtime‚2913vÑç‰ý /loss/FRt1/detection_loss‚0.5494113501419797 .loss/FRt2/detection_loss‚1.304901674299794 0loss/SRqqt1/detection_loss‚1.077419780399196 .loss/FBt2/detection_loss‚1.456043150741607 trainer/global_step‚3399 /loss/FRt2/assignment_loss‚1.034298791191635 2loss/FBt1/assignment_loss‚0.014373228955744007 /loss/FBt1/detection_loss‚0.6896379538032787 $loss/total_loss‚9.83887533608225 epoch‚8 % _timestamp‚1.7842757209443984e+09 /loss/FRt1/assignment_loss‚1.898626786659738 2loss/FBt2/assignment_loss‚0.006073306534593703 1loss/SRqqt2/detection_loss‚0.6447046225974644 _step‚279 2loss/SRqqt1/assignment_loss‚0.6173165984938255 2loss/SRqqt2/assignment_loss‚0.5460680922633933 _runtime‚2913.069728068 ±\ÅŠ‚ 2loss/FBt1/assignment_loss‚0.014373228955744007 /loss/FBt1/detection_loss‚0.6896379538032787 $loss/total_loss‚9.83887533608225 epoch‚8 % _timestamp‚1.7842757209443984e+09 /loss/FRt1/assignment_loss‚1.898626786659738 2loss/FBt2/assignment_loss‚0.006073306534593703 1loss/SRqqt2/detection_loss‚0.6447046225974644 _step‚279 2loss/SRqqt1/assignment_loss‚0.6173165984938255 2loss/SRqqt2/assignment_loss‚0.5460680922633933 _runtime‚2913.069728068 /loss/FRt1/detection_loss‚0.5494113501419797 .loss/FRt2/detection_loss‚1.304901674299794 0loss/SRqqt1/detection_loss‚1.077419780399196 .loss/FBt2/detection_loss‚1.456043150741607 trainer/global_step‚3399 /loss/FRt2/assignment_loss‚1.034298791191635—Dˆ+1“‹‚(:ˆ ”ÆçÒã½ú¤ gpu.0.gpu‚94.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚255.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰? Œ‚(: ”ÆçÒº³‚§ disk.rbd0.in‚0' disk.nvme0c0n1.out‚12138.9501953125 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 172.078125' proc.memory.availableMB‚ 743.2890625 network.sent‚ 5.303131e+06 disk.drbd2165.out‚0 disk.dm-1.out‚ 94.1640625" disk.dm-0.out‚12138.9501953125 cpu‚26.077520033268893 network.recv‚793071$ disk./.usageGB‚545.8415336608887 disk.rbd2.out‚0) proc.memory.percent‚17.01192855834961 disk.drbd2165.in‚0 disk.sda.in‚0 disk.dm-1.in‚0! proc.memory.rssMB‚ 2787.234375 disk.rbd2.in‚0* disk./.usagePercent‚36.631883315534324 disk.sda1.in‚0 disk.sda3.out‚ 94.1640625 disk.rbd1.out‚ 7.74609375 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 172.078125 disk.sda2.in‚0 disk.dm-0.in‚ 172.078125 disk.sda1.out‚0 disk.sda.out‚ 94.1640625$ memory_percent‚95.46332359313965 disk.rbd0.out‚0% disk.nvme0n1.out‚12138.9501953125boºDž‚(:“ £ÆçÒÛ‹¤ gpu.0.gpu‚92.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ¨ý–ÿŽ‚(:ô £ÆçÒÅù£¦ disk.sda.in‚0 disk.dm-1.out‚94.4375 disk.rbd1.out‚ 7.74609375$ memory_percent‚95.46144008636475) proc.memory.percent‚17.01192855834961 cpu‚26.07572070413466 network.sent‚ 5.368138e+06 disk.rbd3.in‚0 disk.nvme0n1.in‚ 172.078125 disk.drbd2165.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚545.8415107727051 disk.nvme0c0n1.in‚ 172.078125 disk.sda.out‚94.4375 disk.dm-1.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚12142.287109375 disk.rbd2.out‚0! proc.memory.rssMB‚ 2787.234375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 network.recv‚797524 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚94.4375$ disk.nvme0n1.out‚12142.287109375 disk.sda1.out‚0& disk.nvme0c0n1.out‚12142.287109375 disk.rbd0.out‚0) disk./.usagePercent‚36.63188177948912 disk.sda3.in‚0 disk.dm-0.in‚ 172.078125 disk.sda1.in‚0( proc.memory.availableMB‚ 743.59765625õÇp2- _wandbruntime‚2952  _runtime‚2952¨{gÉà ' lr-AdamW/pg1‚0.00044500276178388706 ' lr-AdamW/pg2‚0.00044500276178388706 trainer/global_step‚3449 % _timestamp‚1.7842757602730167e+09 _runtime‚2952.399342501 _step‚280x©€Î‘È % _timestamp‚1.7842757602730167e+09 _runtime‚2952.399342501 _step‚280 ' lr-AdamW/pg1‚0.00044500276178388706 ' lr-AdamW/pg2‚0.00044500276178388706 trainer/global_step‚3449˜ ët–2’- _wandbruntime‚2952  _runtime‚2952,s‹|á'“ÛO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚2952.399639501 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7405339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115580473985024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31926556 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3640787601408 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6046291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3449 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59216167 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226146 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3640446262272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚527094714017792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13451627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚524401634076484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520766407487488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31916579 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6046288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523453909377024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524401276487234 _step‚281 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7405625 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119261046976512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520766390448128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31926556 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚527094714017792 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚527094355639296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚523453909377024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27299588 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13451916 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27299590 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27299588 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3680572991488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7842757602737896e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31916579 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976ܬÝüŠ”àO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚527094355639296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚523453909377024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27299588 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13451916 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27299590 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27299588 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3680572991488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7842757602737896e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31916579 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚2952.399639501 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7405339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115580473985024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31926556 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3640787601408 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6046291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3449 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59216167 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226146 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3640446262272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚527094714017792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13451627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚524401634076484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚520766407487488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31916579 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6046288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523453909377024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524401276487234 _step‚281 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7405625 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119261046976512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520766390448128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31926556 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚527094714017792 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13455377 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115613902743552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115613921355776 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31925631 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524548798212828 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520912901120000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3641800235520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚524549155802190 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚527242967681536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3641800235520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27307222 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523601167446016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523601167446016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6048036 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119295612498432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2953.012519533 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7407620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523601150406656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842757608866973e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27307222 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7407344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3636237642830 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚282 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31935622 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31925631 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3635897092828 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6048033 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27307220 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚524549155802190 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚527242967681536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3641800235520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27307222 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523601167446016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523601167446016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6048036 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119295612498432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2953.012519533 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7407620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523601150406656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842757608866973e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27307222 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7407344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3636237642830 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚282 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31935622 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31925631 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3635897092828 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6048033 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27307220 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13455377 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115613902743552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115613921355776 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31925631 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524548798212828 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520912901120000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3641800235520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968š£Õ2à2˜- _wandbruntime‚2953  _runtime‚2953âɲÿ™þ 1loss/SRqqt1/assignment_loss‚0.648447377990769 %loss/total_loss‚9.509763593627353 epoch‚8 _runtime‚2953.014322367 /loss/FRt1/assignment_loss‚1.887575371824082 /loss/FBt1/detection_loss‚0.4965136675957452 2loss/FBt1/assignment_loss‚0.024667855967097474 1loss/FBt2/assignment_loss‚0.01768262922248596 .loss/FRt2/detection_loss‚1.356976339113605 1loss/SRqqt1/detection_loss‚0.9816523912011094 /loss/FBt2/detection_loss‚1.0735084842145444 trainer/global_step‚3449 $ _timestamp‚1.784275760888926e+09 _step‚283 2loss/SRqqt2/assignment_loss‚0.6461092276072321 /loss/FRt1/detection_loss‚0.5428340707944164 1loss/SRqqt2/detection_loss‚0.6671194838878285 0loss/FRt2/assignment_loss‚1.1666766942084372V“š šƒ epoch‚8 _runtime‚2953.014322367 /loss/FRt1/assignment_loss‚1.887575371824082 /loss/FBt1/detection_loss‚0.4965136675957452 2loss/FBt1/assignment_loss‚0.024667855967097474 1loss/FBt2/assignment_loss‚0.01768262922248596 .loss/FRt2/detection_loss‚1.356976339113605 1loss/SRqqt1/detection_loss‚0.9816523912011094 /loss/FBt2/detection_loss‚1.0735084842145444 trainer/global_step‚3449 $ _timestamp‚1.784275760888926e+09 _step‚283 2loss/SRqqt2/assignment_loss‚0.6461092276072321 /loss/FRt1/detection_loss‚0.5428340707944164 1loss/SRqqt2/detection_loss‚0.6671194838878285 0loss/FRt2/assignment_loss‚1.1666766942084372 1loss/SRqqt1/assignment_loss‚0.648447377990769 %loss/total_loss‚9.509763593627353›æZž›‚(:“ ²ÆçÒÝÿ’¤ gpu.0.gpu‚62.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.162' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.432399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ópßœ‚(:Š ²ÆçÒŠí¦ disk.nvme0n1.in‚ 172.79296875 disk.rbd0.out‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 172.79296875( disk.nvme0c0n1.out‚12151.43115234375 disk.drbd2165.in‚0 disk.dm-1.out‚ 94.6015625( proc.memory.availableMB‚ 743.60546875 network.sent‚ 5.384143e+06 disk.sda1.out‚0 disk.sda.out‚ 94.6015625 disk.dm-1.in‚0# disk.dm-0.out‚12151.43115234375 disk.sda2.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.01192855834961 network.recv‚800914* disk./.usagePercent‚36.631879731428846 disk.sda1.in‚0 disk.sda3.out‚ 94.6015625$ memory_percent‚95.46139240264893! proc.memory.rssMB‚ 2787.234375& disk.nvme0n1.out‚12151.43115234375 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚26.073915209727772 disk.rbd2.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 7.87109375 disk.rbd3.out‚0 disk.dm-0.in‚ 172.79296875 proc.cpu.threads‚17# disk./.usageGB‚545.841480255127 disk.rbd2.out‚0{†“‚(:ˆ ÁÆçҷͤ gpu.0.gpu‚65.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«(àŒž‚(:Š ÁÆçÒ€«æ¦) proc.memory.percent‚17.01192855834961 cpu‚26.072222082646153 disk.dm-1.out‚ 94.8359375& disk.nvme0n1.out‚12158.42041015625 disk.rbd0.in‚0 disk.sda3.out‚ 94.8359375$ memory_percent‚95.46101093292236 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12158.42041015625! proc.memory.rssMB‚ 2787.234375 disk.rbd1.out‚ 7.87109375 disk.rbd3.in‚0 disk.sda2.in‚0 network.recv‚805566 disk.dm-0.in‚ 173.62109375 disk.rbd2.out‚0 disk.nvme0n1.in‚ 173.62109375( proc.memory.availableMB‚ 743.66796875$ disk./.usageGB‚545.8414688110352 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚12158.42041015625 disk.rbd2.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 94.8359375 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 5.43426e+06* disk./.usagePercent‚36.631878963406244 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 173.62109375 |½”’Ÿ‚(:‡ ÐÆçÒàñ¥ gpu.0.gpu‚90.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚205.035' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ü+o ‚(:Š ÐÆçÒ¢™™§ disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 7.93359375 disk.rbd2.out‚0& disk.nvme0n1.out‚12167.04052734375 disk.sda1.in‚0( disk.nvme0c0n1.out‚12167.04052734375 disk.sda.out‚ 95.03515625( proc.memory.availableMB‚ 678.87890625) disk./.usagePercent‚36.63187768336857 disk.sda3.out‚ 95.03515625 disk.rbd1.in‚0! proc.memory.rssMB‚ 2787.234375) proc.memory.percent‚17.01192855834961 disk.rbd0.in‚0 cpu‚26.070454948413822 disk.rbd0.out‚0 disk.dm-1.out‚ 95.03515625 disk.dm-0.in‚ 174.3359375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 network.recv‚808545$ disk./.usageGB‚545.8414497375488 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 5.441419e+06 disk.rbd3.in‚0 disk.nvme0n1.in‚ 174.3359375! disk.nvme0c0n1.in‚ 174.3359375$ memory_percent‚95.85645198822021# disk.dm-0.out‚12167.04052734375 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0y¼â52¡- _wandbruntime‚2992  _runtime‚2992í%ø*Ä¢¾ % lr-AdamW/pg1‚0.000444106840228988 % lr-AdamW/pg2‚0.000444106840228988 trainer/global_step‚3499 $ _timestamp‚1.784275800024736e+09 _runtime‚2992.150427935 _step‚2842ì'ɣà % lr-AdamW/pg1‚0.000444106840228988 % lr-AdamW/pg2‚0.000444106840228988 trainer/global_step‚3499 $ _timestamp‚1.784275800024736e+09 _runtime‚2992.150427935 _step‚284œå%Iê2¤- _wandbruntime‚2992  _runtime‚2992i¸–Ûé¥ÚO BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚534507461001216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚530815977897984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530815977897984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32381008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F(‹Z½÷ @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3691141764096 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27681190 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60062198 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7508250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7507988 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3685502526014 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚2992.151681071 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530815960858624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27681188 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27681188 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120983696874496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117251930523648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528091941081088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚534507102622720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3731784963072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚285 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531777426567742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3691141764096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27681190 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3685843075904 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32381008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534507102622720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13641791 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚530815960858624k (rå'¦ßO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3691141764096 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27681190 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60062198 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7508250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7507988 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3685502526014 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚2992.151681071 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530815960858624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27681188 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27681188 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120983696874496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117251930523648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528091941081088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚534507102622720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3731784963072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚285 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531777426567742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3691141764096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27681190 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3685843075904 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32381008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534507102622720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13641791 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚530815960858624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚534507461001216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚530815977897984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530815977897984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32381008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0‘RР2§- _wandbruntime‚2992  _runtime‚2992„P"é&¨ãM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528238434713600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27688820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117285340669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3732894858752 _step‚286 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚534655356279296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534655714664960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3692495737344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3692495737344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3692154391040 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13645308 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534655714664960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚530963201888256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7510025 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068903 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32390074 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6135286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531925305882698 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60078896 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068903 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3692154391040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚530963201888256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32390074 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32380083 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _runtime‚2992.763673772 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚530963218927616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528238451752960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27688822 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121018149056000 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3686513579736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27688820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262G¶Ájæ©èM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60078896 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068903 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3692154391040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚530963201888256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32390074 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32380083 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _runtime‚2992.763673772 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚530963218927616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528238451752960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27688822 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121018149056000 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/alloc/šFUated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3686513579736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27688820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528238434713600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27688820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117285340669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3732894858752 _step‚286 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚534655356279296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚534655714664960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3692495737344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3692495737344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3692154391040 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13645308 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534655714664960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚530963201888256 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7510025 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068903 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32390074 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6135286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚531925305882698ž¦œÞ2ª- _wandbruntime‚2992  _runtime‚2992 _Ï«€ _runtime‚2992.766224679 _step‚287 2loss/SRqqt2/assignment_loss‚0.5944039988467675 2loss/FBt2/assignment_loss‚0.032732505803866094 /loss/FBt1/detection_loss‚0.5599048842710478 2loss/SRqqt1/assignment_loss‚0.5993173928641263 1loss/SRqqt1/detection_loss‚1.0387734715501225 /loss/FBt2/detection_loss‚0.7006745193282078 0loss/FRt1/assignment_loss‚1.9222501525634648 0loss/FRt2/assignment_loss‚1.1381586959305983 1loss/FBt1/assignment_loss‚0.06785181659136083 -loss/FRt2/detection_loss‚1.34825565571986 1loss/SRqqt2/detection_loss‚0.7213129259480802 trainer/global_step‚3499 % _timestamp‚1.7842758006408877e+09 /loss/FRt1/detection_loss‚0.5516819182838356 %loss/total_loss‚9.275317937701338 epoch‚8†ÑQÁ ¬… _runtime‚2992.766224679 _step‚287 2loss/SRqqt2/assignment_loss‚0.5944039988467675 2loss/FBt2/assignment_loss‚0.032732505803866094 /loss/FBt1/detection_loss‚0.5599048842710478 2loss/SRqqt1/assignment_loss‚0.5993173928641263 1loss/SRqqt1/detection_loss‚1.0387734715501225 /loss/FBt2/detection_loss‚0.7006745193282078 0loss/FRt1/assignment_loss‚1.9222501525634648 0loss/FRt2/assignment_loss‚1.1381586959305983 1loss/FBt1/assignment_loss‚0.06785181659136083 -loss/FRt2/detection_loss‚1.34825565571986 1loss/SRqqt2/detection_loss‚0.7213129259480802 trainer/global_step‚3499 % _timestamp‚1.7842758006408877e+09 /loss/FRt1/detection_loss‚0.5516819182838356 %loss/total_loss‚9.275317937701338 epoch‚8Ÿ- “­‚(:’ ßÆçÒ•¦ƒ¦ gpu.0.gpu‚94.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.603' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.12060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0] ÷ஂ(:… ßÆçÒ–Âû¨$ disk./.usageGB‚545.8383865356445 disk.dm-0.in‚ 175.1640625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 95.3828125$ memory_percent‚96.96691036224365 disk.rbd3.out‚0 disk.sda.out‚ 95.3828125 disk.sda3.in‚0 disk.dm-1.out‚ 95.3828125) proc.memory.percent‚17.01195240020752 disk.dm-0.out‚ 12173.5078125 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd2.out‚0$ disk.nvme0c0n1.out‚ 12173.5078125 disk.dm-1.in‚0 disk.rbd1.out‚ 8.08203125# proc.memory.rssMB‚ 2787.23828125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 496.94140625 cpu‚26.06859880300821 disk.sda1.in‚0 disk.drbd2165.out‚0 network.recv‚811889) disk./.usagePercent‚36.63167210931857" disk.nvme0n1.out‚ 12173.5078125 disk.sda1.out‚0 disk.sda2.in‚0 network.sent‚ 5.444383e+06 disk.rbd2.in‚0 disk.nvme0n1.in‚ 175.1640625! disk.nvme0c0n1.in‚ 175.1640625 disk.rbd1.in‚ 0.01171875㣭ÿ‘¯‚(:† îÆçÒ¦ ¤ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öýà#°‚(:˜ îÆçҎ覦 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 95.55078125 disk.rbd1.out‚28.22802734375# proc.memory.rssMB‚ 2787.23828125 cpu‚26.066921455623884 disk.nvme0n1.in‚ 175.1640625 disk.sda1.in‚0 disk.sda.out‚ 95.55078125 disk.rbd3.out‚0& disk.nvme0n1.out‚12176.29833984375) proc.memory.percent‚17.01195240020752) disk./.usagePercent‚36.63168260562748 disk.dm-1.in‚0 disk.rbd3.in‚0$ memory_percent‚96.96667194366455( proc.memory.availableMB‚ 496.98046875 proc.cpu.threads‚17 disk.rbd2.out‚0 disk.sda1.out‚0$ disk./.usageGB‚545.8385429382324 disk.dm-0.in‚ 175.1640625# disk.dm-0.out‚12176.29833984375 disk.rbd2.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 175.1640625 network.sent‚ 5.50741e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875 network.recv‚816105( disk.nvme0c0n1.out‚12176.29833984375 disk.sda2.out‚0 disk.dm-1.out‚ 95.55078125§¥ž±‚(:“ ýÆçÒ¼™£ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wS8²‚(:‘ ýÆçÒÔ÷®¦ network.sent‚ 5.513665e+06 disk.rbd2.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12199.30908203125 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875# disk.dm-0.out‚12199.30908203125 disk.rbd2.out‚0 disk.sda2.in‚0 disk.rbd1.out‚90.94287109375# proc.memory.rssMB‚ 2787.23828125 network.recv‚818829 disk.dm-0.in‚ 175.87890625$ memory_percent‚96.96516990661621 cpu‚26.06521206330958) disk./.usagePercent‚36.64870608262432 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚95.71875 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.sda.out‚95.71875 disk.sda3.out‚95.71875 disk.dm-1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 175.87890625& disk.nvme0n1.out‚12199.30908203125$ disk./.usageGB‚546.0922050476074 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 175.87890625 disk.sda.in‚0' proc.memory.availableMB‚ 497.2265625 disk.sda1.out‚0) proc.memory.percent‚17.01195240020752,0Ñ2³- _wandbruntime‚3032  _runtime‚3032¥5øÉ´Ã ' lr-AdamW/pg1‚0.00044319757381968637 ' lr-AdamW/pg2‚0.00044319757381968637 trainer/global_step‚3549 % _timestamp‚1.7842758399526768e+09 _runtime‚3032.078384879 _step‚288¤UmŽÎµÈ _step‚288 ' lr-AdamW/pg1‚0.00044319757381968637 ' lr-AdamW/pg2‚0.00044319757381968637 trainer/global_step‚3549 % _timestamp‚1.7842758399526768e+09 _runtime‚3032.078384879 €µ›2¶- _wandbruntime‚3032  _runtime‚3032·Ò²à'·ÚO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚538178029379584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13831638 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538178029379584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32835460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32825483 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚3549 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541920207984640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782989817856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6220791 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122706339655680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3741837265920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539153934237500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚3032.083358591 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539153576648250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60888271 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚538178012340224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28062788 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3742178605056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60898250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28062788 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541919849606144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13831375 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚289 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28062790 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538178012340224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842758399538352e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32825483 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚3549 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541920207984640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782989817856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6220791 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122706339655680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3741837265920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539153934237500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚3032.083358591 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539153576648250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60888271 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚538178012340224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28062788 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3742178605056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60898250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28062788 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541919849606144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13831375 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚289 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28062790 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538178012340224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842758399538352e+09 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542068103262720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3784102703616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32834535 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32834535 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122740899912192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539301098373844 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539301455963206 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28070420 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13835196 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914948 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28070422 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚290 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚538325270409216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538325253369856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32844526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0:äOï&»éM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13835464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60914948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7612928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3737130066644 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3742849892864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118956778596352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542068103262720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3784102703616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32834535 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32834535 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122740899912192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539301098373844 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539301455963206 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28070420 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13835196 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914948 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28070422 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚290 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚538325270409216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538325253369856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32844526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28444390 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3792874106880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3792532767744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546529726728758 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545540080861184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61734302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33279935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724323 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7713639 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545540080861184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549332596589568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120594806376448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542742991228928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28444388 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _runtime‚3072.066802867 nâ;æŽ G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6308041 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542743008268288 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28444388 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6308038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124429100517888 % _timestamp‚1.7842758799409397e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33289912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549332596589568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120594806376448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542742991228928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28444388 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _runtime‚3072.066802867 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6308041 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542743008268288 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28444388 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6308038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124429100517888 % _timestamp‚1.7842758799409397e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33289912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28444390 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3792874106880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3792532767744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546529726728758 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545540080861184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61734302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33279935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724323 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7713639 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545540080861184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542889518940160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚294 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14025133 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3788087103554 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3793545394688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14025388 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28452020 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61751000 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28452022 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7715350 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3835317278208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3787746553552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚545687321890816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3793886740992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741007 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28452022 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3072.679784936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚542889501900800 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚549481208631808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚549480850246144 trainer/global_step‚3599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28452020 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0&RïO&ËèM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120628235134976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741007 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚545687304851456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546677606043714 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545687304851456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33288987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7715602 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚549481208631808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚549480850246144 trainer/global_step‚3599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28452020 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6309783 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120628216522752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 $ _timestamp‚1.784275880553958e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124463449425408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542889518940160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚294 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14025133 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3788087103554 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3793545394688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14025388 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28452020 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61751000 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28452022 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7715350 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3835317278208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3787746553552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚545687321890816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3793886740992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741007 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28452022 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3072.679784936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚542889501900800 F&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6454426050186157 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6425588130950928 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.3175564110279083 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.022540738806128502 *jet/accuracy_1_of_4‚0.9659627079963684 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9903906583786011 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1685868799686432 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17238907516002655 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38011443614959717 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5742121934890747 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5554166436195374 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2045743316411972 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6401357650756836 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6468610763549805 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20012348890304565 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9000637531280518 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940221905708313 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 /particle/accuracy_3_of_4‚0.9965986609458923 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6886001825332642 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6864622235298157 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03806929290294647 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3064941465854645 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6977569460868835 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9935814738273621 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7796562910079956 *jet/accuracy_2_of_4‚0.7287625074386597 _runtime‚3089.729863806 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8609652519226074 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6449728012084961 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4972163736820221 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7363912463188171 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08833859860897064 particle/accuracy_1_of_1‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4570396840572357 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.692743718624115 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04210478812456131 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6836539506912231 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922599792480469 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7386320233345032 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1924671232700348 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7814360857009888 +particle/specificity‚0.9054917097091675 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5686091184616089 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972034692764282 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1690085381269455 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 6validation_average_jet_accuracy‚0.3607219159603119 particle/accuracy_5_of_5‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7676190137863159 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7492805123329163 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9882417321205139 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11069773137569427 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.658050000667572 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9849570989608765 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850418567657471 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7386202812194824 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4791358411312103 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6368693113327026 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6864784955978394 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026289531961083412 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09334615617990494 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020648645237088203 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18290194869041443 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9764834642410278 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011839286424219608 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9190783500671387 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7649990916252136 particle/accuracy_1_of_2‚1 _step‚296 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚3599 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04210478812456131 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860069751739502 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_3‚0.07257472723722458 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.824508786201477 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26848769187927246 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37699228525161743 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17712384462356567 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8325586915016174 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18040773272514343 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6203850507736206 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6169146299362183 )loss/val_total_loss‚8.937945926975924 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9931291937828064 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8265683054924011 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9916468262672424 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8655898571014404 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20012348890304565 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_2_of_2‚0.1423773318529129 particle/accuracy_1_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4118324816226959 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4765625 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2131449431180954 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2648012936115265 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6454426050186157 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2244747430086136 epoch‚8 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1819317489862442 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19034597277641296 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6286977529525757 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9214373826980591 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884788990020752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898289442062378 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9166790246963501 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21423104405403137 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9961310029029846 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16226446628570557 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9457000494003296 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8103553056716919 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948102235794067 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6668070554733276 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6390753388404846 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1525000035762787 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5756232738494873 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6335452198982239 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.043928079307079315 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3063509166240692 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40008875727653503 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12454897910356522 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9961310029029846 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7354313731193542 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7657870054244995 (particle/accuracy‚0.8460479974746704 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6436585187911987 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7211108207702637 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7263027429580688 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4930555522441864 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04365108534693718 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6852215528488159 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7220710515975952 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21262633800506592 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03929334878921509 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1872115582227707 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20768867433071136 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04408058151602745 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 *jet/accuracy_2_of_3‚0.5369212627410889 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7842758976033497e+09 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1508210450410843 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7797608375549316 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10646580159664154 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 /particle/accuracy_3_of_3‚0.9994933605194092 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48281365633010864 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.044236406683921814 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7343467473983765 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8042628169059753 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0uÔ×/5ÓÏ{ ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.658050000667572 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9849570989608765 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850418567657471 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7386202812194824 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4791358411312103 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6368693113327026 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6864784955978394 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026289531961083412 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09334615617990494 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020648645237088203 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18290194869041443 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9764834642410278 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011839286424219608 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9190783500671387 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7649990916252136 particle/accuracy_1_of_2‚1 _step‚296 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚3599 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04210478812456131 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860069751739502 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_3‚0.07257472723722458 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.824508786201477 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26848769187927246 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37699228525161743 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17712384462356567 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8325586915016174 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18040773272514343 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6203850507736206 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6169146299362183 )loss/val_total_loss‚8.937945926975924 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9931291937828064 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8265683054924011 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9916468262672424 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8655898571014404 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20012348890304565 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_2_of_2‚0.1423773318529129 particle/accuracy_1_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4118324816226959 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4765625 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2131449431180954 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2648012936115265 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6454426050186157 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2244747430086136 epoch‚8 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1819317489862442 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19034597277641296 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6286977529525757 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9214373826980591 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884788990020752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898289442062378 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9166790246963501 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21423104405403137 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9961310029029846 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16226446628570557 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9457000494003296 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8103553056716919 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948102235794067 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6668070554733276 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6390753388404846 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1525000035762787 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5756232738494873 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6335452198982239 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.043928079307079315 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3063509166240692 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40008875727653503 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12454897910356522 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9961310029029846 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7354313731193542 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7657870054244995 (particle/accuracy‚0.8460479974746704 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6436585187911987 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7211108207702637 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7263027429580688 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4930555522441864 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04365108534693718 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6852215528488159 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7220710515975952 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21262633800506592 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03929334878921509 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1872115582227707 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20768867433071136 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04408058151602745 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 *jet/accuracy_2_of_3‚0.5369212627410889 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7842758976033497e+09 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1508210450410843 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7797608375549316 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10646580159664154 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 /particle/accuracy_3_of_3‚0.9994933605194092 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48281365633010864 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.044236406683921814 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7343467473983765 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8042628169059753 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20057609677314758 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6483075022697449 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7932729125022888 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8042628169059753 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959671497344971 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6428982615470886 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9942865967750549 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6429803967475891 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6454426050186157 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6425588130950928 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.3175564110279083 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.022540738806128502 *jet/accuracy_1_of_4‚0.9659627079963684 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9903906583786011 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1685868799686432 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17238907516002655 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38011443614959717 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5742121934890747 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5554166436195374 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2045743316411972 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6401357650756836 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6468610763549805 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20012348890304565 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9000637531280518 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940221905708313 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 /particle/accuracy_3_of_4‚0.9965986609458923 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6886001825332642 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6864622235298157 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03806929290294647 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3064941465854645 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6977569460868835 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9935814738273621 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7796562910079956 *jet/accuracy_2_of_4‚0.7287625074386597 _runtime‚3089.729863806 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8609652519226074 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6449728012084961 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4972163736820221 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/1FRt*SRq”×éѦqt1FBt/SRqqt_purity‚0.7363912463188171 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08833859860897064 particle/accuracy_1_of_1‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4570396840572357 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.692743718624115 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04210478812456131 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6836539506912231 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922599792480469 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7386320233345032 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1924671232700348 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7814360857009888 +particle/specificity‚0.9054917097091675 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5686091184616089 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972034692764282 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1690085381269455 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 6validation_average_jet_accuracy‚0.3607219159603119 particle/accuracy_5_of_5‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7676190137863159 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7492805123329163 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9882417321205139 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11069773137569427¨ªùsœÔ‚(:‘ ¹ÇçÒ…š¿£ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÿ“™ Õ‚(:‚ ¹ÇçÒÝÚù§$ disk./.usageGB‚546.0929222106934 disk.rbd2.in‚0 disk.nvme0n1.in‚178.25 disk.sda2.in‚0) disk./.usagePercent‚36.64875421204076 disk.dm-0.in‚178.25 disk.nvme0c0n1.in‚178.25 disk.sda.in‚0) proc.memory.percent‚17.06836223602295 network.recv‚834580 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0# proc.memory.availableMB‚487.625" disk.dm-0.out‚12477.2158203125' disk.nvme0c0n1.out‚12477.2158203125 disk.sda.out‚ 96.67578125 disk.rbd1.in‚ 0.01171875 network.sent‚ 5.660463e+06 disk.sda2.out‚0 disk.sda3.out‚ 96.67578125 disk.rbd1.out‚91.31005859375$ memory_percent‚97.02377319335938# proc.memory.rssMB‚ 2796.48046875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚12477.2158203125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 96.67578125 proc.cpu.threads‚17 disk.rbd2.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 cpu‚26.080042344701234ÒL{ö’Ö‚(:‡ ÈÇçÒÈ‹»¥ gpu.0.gpu‚83.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.297 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K-´ã ׂ(:• ÈÇçÒÑž¾§) disk./.usagePercent‚36.64876138025171 disk.dm-1.out‚ 97.0234375 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 178.96484375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚838918$ disk./.usageGB‚546.0930290222168" disk.nvme0c0n1.in‚ 178.96484375 disk.drbd2165.out‚0& proc.memory.availableMB‚ 529.015625# proc.memory.rssMB‚ 2755.34765625 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875 disk.drbd2165.in‚0% disk.nvme0n1.out‚12483.8447265625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 97.0234375 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚12483.8447265625 cpu‚26.078320235866364 network.sent‚ 5.707106e+06' disk.nvme0c0n1.out‚12483.8447265625 disk.sda3.in‚0 disk.dm-0.in‚ 178.96484375$ memory_percent‚96.77114486694336 disk.sda3.out‚ 97.0234375 disk.rbd1.out‚91.41943359375* proc.memory.percent‚16.817307472229004 disk.sda.in‚0 disk.sda2.in‚0=˜ØžØ‚(:“ ×ÇçÒÿü㢠gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚252.395' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.479000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù¹î Ù‚(:• ×ÇçÒÝÜë¦) disk./.usagePercent‚36.64879952537432 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 97.3671875 disk.rbd3.out‚0# proc.memory.rssMB‚ 2755.34765625% disk.nvme0n1.out‚12492.9306640625' disk.nvme0c0n1.out‚12492.9306640625" disk.dm-0.out‚12492.9306640625* proc.memory.percent‚16.817307472229004 cpu‚26.07669673952149 disk.rbd0.out‚0 disk.sda.out‚ 97.3671875 disk.rbd2.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0$ memory_percent‚96.77109718322754 proc.cpu.threads‚17 disk.nvme0n1.in‚ 179.79296875 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875" disk.nvme0c0n1.in‚ 179.79296875 disk.rbd1.out‚91.41943359375 disk.rbd2.out‚0' proc.memory.availableMB‚ 529.0234375 network.sent‚ 5.712298e+06 network.recv‚842377 disk.drbd2165.out‚0 disk.dm-1.in‚0$ disk./.usageGB‚546.0935974121094 disk.dm-1.out‚ 97.3671875 disk.dm-0.in‚ 179.79296875EUf/2Ú- _wandbruntime‚3129  _runtime‚3129ÉÊ2ÜÇÛÁ & lr-AdamW/pg1‚0.0004413392408931687 & lr-AdamW/pg2‚0.0004413392408931687 trainer/global_step‚3649 % _timestamp‚1.7842759369687119e+09 _runtime‚3129.095467107 _step‚297Ö‹m"ÌÜÆ & lr-AdamW/pg1‚0.0004413392408931687 & lr-AdamW/pg2‚0.0004413392408931687 trainer/global_step‚3649 % _timestamp‚1.7842759369687119e+09 _runtime‚3129.095467107 _step‚297©¾” 52Ý- _wandbruntime‚3129  _runtime‚3129Ü%€à'ÞÚO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126185834770944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558692643428864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554831698993152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚3129.095609086 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3861302814208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3861302814208 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855412261857 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751363 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33804709 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6397808 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚555832259748575 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554831681953792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6397811 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3903589791232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚555832617337825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558692643428864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814686 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14245047 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541Áût-å'ßßO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126185727581696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122282156402688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7842759369694462e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28946656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14245322 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚551977205075968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚558693001807360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚3649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33804709 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6397808 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚555832259748575 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554831681953792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6397811 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3903589791232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚555832617337825 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558692643428864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814686 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14245047 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126185834770944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558692643428864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554831698993152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚3129.095609086 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3861302814208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3861302814208 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855412261857 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751363 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 I›ŸêáãM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122315585161216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3856082765689 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚552123715747840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126220287076352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6399553 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28954288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28954286 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33823752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3862315448320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3904701915136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28954288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7842759375813375e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554978940022784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554978940022784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14248859 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861974102016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62768047 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555980139063531 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚558841255471104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28954286 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6399556 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778040 OGDeviceStatsMonitor.on_train_b†Ð¿7ÿatch_end/inactive_split.small_pool.current‚263 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3856423315691 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558841255471104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7849569 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122315566548992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7849306 _runtime‚3129.707159584 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555979781474169 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862315448320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554978922983424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3904621733888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14249125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33823752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3649 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62768047 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126220188282880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888hwî&âèM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126220287076352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6399553 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28954288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28954286 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33823752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3862315448320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3904701915136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28954288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7842759375813375e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554978940022784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554978940022784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14248859 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861974102016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62768047 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555980139063531 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚558841255471104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28954286 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6399556 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3856423315691 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558841255471104 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7849569 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122315566548992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7849306 _runtime‚3129.707159584 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555979781474169 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862315448320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554978922983424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3904621733888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14249125 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33823752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3649 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62768047 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126220188282880 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122315585161216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3856082765689 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚552123715747840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0«Žuhs2ã- _wandbruntime‚3129  _runtime‚3129ÔQäþ 2loss/SRqqt1/assignment_loss‚0.6051819401327521 0loss/FBt1/assignment_loss‚0.0691722510744105 %loss/total_loss‚9.284892709213196 $ _timestamp‚1.784275937584508e+09 /loss/FRt2/detection_loss‚1.3300675858008257 /loss/FBt2/detection_loss‚1.0056048046099022 trainer/global_step‚3649 0loss/FRt2/assignment_loss‚1.0317285271321057 1loss/FBt2/assignment_loss‚0.02371248918538349 1loss/SRqqt1/detection_loss‚1.0457029227431105 1loss/SRqqt2/detection_loss‚0.6368102992292423 /loss/FBt1/detection_loss‚0.5767068648138313 epoch‚9 _runtime‚3129.709840606 _step‚300 /loss/FRt1/assignment_loss‚1.835411504498178 2loss/SRqqt2/assignment_loss‚0.5837072944508968 /loss/FRt1/detection_loss‚0.5410862255425578Šî´ åƒ 0loss/FRt2/assignment_loss‚1.0317285271321057 1loss/FBt2/assignment_loss‚0.02371248918538349 1loss/SRqqt1/detection_loss‚1.0457029227431105 1loss/SRqqt2/detection_loss‚0.6368102992292423 /loss/FBt1/detection_loss‚0.5767068648138313 epoch‚9 _runtime‚3129.709840606 _step‚300 /loss/FRt1/assignment_loss‚1.835411504498178 2loss/SRqqt2/assignment_loss‚0.5837072944508968 /loss/FRt1/detection_loss‚0.5410862255425578 2loss/SRqqt1/assignment_loss‚0.6051819401327521 0loss/FBt1/assignment_loss‚0.0691722510744105 %loss/total_loss‚9.284892709213196 $ _timestamp‚1.784275937584508e+09 /loss/FRt2/detection_loss‚1.3300675858008257 /loss/FBt2/detection_loss‚1.0056048046099022 trainer/global_step‚3649¬Ýg”¦“æ‚(:ˆ æÇçÒ—ºÑ¤ gpu.0.gpu‚98.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š—£ç‚(:” æÇçÒ»×̦ disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 180.5078125 disk.rbd2.out‚0! disk.nvme0c0n1.in‚ 180.5078125 disk.sda2.out‚0 disk.dm-1.out‚ 97.69921875' proc.memory.availableMB‚ 529.6953125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 97.69921875 disk.rbd1.out‚91.48193359375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875! disk.dm-0.out‚12499.837890625 disk.drbd2165.out‚0$ disk./.usageGB‚546.0906982421875 disk.sda.in‚0$ memory_percent‚96.76699638366699* proc.memory.percent‚16.817307472229004 cpu‚26.075002749254526 network.sent‚ 5.715321e+06 disk.rbd0.out‚0& disk.nvme0c0n1.out‚12499.837890625 disk.sda3.in‚0 network.recv‚845567 disk.nvme0n1.in‚ 180.5078125 disk.sda1.in‚0 disk.sda3.out‚ 97.69921875# proc.memory.rssMB‚ 2755.34765625$ disk.nvme0n1.out‚12499.837890625 disk.dm-1.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.648604959648296]“»›è‚(: õÇçÒÙÿÿ£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚291.631' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.32619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L|¼%é‚(:š õÇçÒ©þ‰¦ disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 97.87890625& disk.nvme0n1.out‚12506.63525390625 disk.rbd0.in‚0 disk.sda.out‚ 97.87890625 disk.rbd1.in‚ 0.01171875 network.sent‚ 5.780464e+06 disk.rbd0.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 180.5078125# disk.dm-0.out‚12506.63525390625 disk.rbd2.in‚0$ disk./.usageGB‚546.0950889587402 disk.sda1.in‚0# proc.memory.rssMB‚ 2755.34765625* proc.memory.percent‚16.817307472229004) disk./.usagePercent‚36.64889962432021 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 180.5078125 cpu‚26.073408094977534 network.recv‚850075 disk.nvme0n1.in‚ 180.5078125( disk.nvme0c0n1.out‚12506.63525390625 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 97.87890625 disk.dm-1.in‚0 disk.rbd1.out‚91.48193359375 disk.rbd2.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0$ memory_percent‚96.76663875579834( proc.memory.availableMB‚ 529.75390625ˆV“ê‚(:ˆ „ÈçÒ‰Êú¢ gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|µÚë‚(:‹ „ÈçÒÕ‰¾§ network.recv‚853547 disk.rbd2.in‚0% disk.nvme0n1.out‚12514.1220703125! disk.nvme0c0n1.in‚ 181.3359375 network.sent‚ 5.785528e+06 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚96.76792621612549 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0) disk./.usagePercent‚36.64889220010171 disk.sda.out‚98.0625 disk.sda3.out‚98.0625 disk.dm-1.out‚98.0625" disk.dm-0.out‚12514.1220703125 disk.nvme0n1.in‚ 181.3359375 cpu‚26.071717276993965 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2755.34765625$ disk./.usageGB‚546.0949783325195 disk.dm-0.in‚ 181.3359375 disk.rbd2.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚91.56005859375' disk.nvme0c0n1.out‚12514.1220703125 disk.rbd1.in‚ 0.01171875( proc.memory.availableMB‚ 529.54296875* proc.memory.percent‚16.817307472229004¬2ì- _wandbruntime‚3168  _runtime‚3168" Ž‘Éíà _runtime‚3168.865763015 _step‚301 ' lr-AdamW/pg1‚0.00044039029373320397 ' lr-AdamW/pg2‚0.00044039029373320397 trainer/global_step‚3699 % _timestamp‚1.7842759767401783e+09ƒ>"SÎîÈ ' lr-AdamW/pg1‚0.00044039029373320397 ' lr-AdamW/pg2‚0.00044039029373320397 trainer/global_step‚3699 % _timestamp‚1.7842759767401783e+09 _runtime‚3168.865763015 _step‚301­pd‰æ2ï- _wandbruntime‚3168  _runtime‚3168fþ,;à'ðÚO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127908538659840 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6485061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3911998316032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚566105390412288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3911656976896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚562193733435392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566105390412288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3911656976896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3905688198875 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7952197 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29328254 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29328256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562193733435392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚559302721630208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _runtime‚3168.866987226 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34259161 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚563208409829083 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚559302738669568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14437536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚3699 _step‚302 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587415 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123953575716864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34259161 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6485058 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3906028748765 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562193750474752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29328256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14437255 % _timestamp‚1.7842759767412221e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562193750474752 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3954962942976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597394 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34269138 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911998316032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127908652140544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123953594329088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚566105748790784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.smalÓÜS„~#l_pool.current‚9977 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123953594329088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚566105748790784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚566105390412288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3911656976896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚562193733435392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566105390412288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3911656976896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3905688198875 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7952197 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29328254 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29328256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562193733435392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚559302721630208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _runtime‚3168.866987226 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34259161 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚563208409829083 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚559302738669568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14437536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚3699 _step‚302 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587415 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123953575716864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34259161 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6485058 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3906028748765 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562193750474752 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29328256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14437255 % _timestamp‚1.7842759767412221e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562193750474752 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3954962942976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169®_G2ò- _wandbruntime‚3169  _runtime‚3169¸œUßê&óäM % _timestamp‚1.7842759773552635e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7954306 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6486806 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562340974465024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29335888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562340991504384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3913010950144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562340974465024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614092 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127942983501312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14441376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚566254002454528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3913010950144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6486803 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34278204 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14441109 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562340974465024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614092 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127942983501312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14441376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚566254002454528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3913010950144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6486803 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34278204 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14441109 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FèÏ¡û‚(:– ¢ÈçÒÃÇ¡¦ disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚91.62255859375" disk.nvme0c0n1.in‚ 182.87890625# memory_percent‚96.7677116394043‹·aJ€& proc.memory.availableMB‚ 529.578125 disk.rbd3.in‚0 disk.dm-0.in‚ 182.87890625 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 98.8046875 network.recv‚861512 disk.rbd2.in‚0( disk.nvme0c0n1.out‚12531.84228515625 disk.sda2.out‚0( disk./.usagePercent‚36.6491620320428 disk.dm-1.in‚0# disk.dm-0.out‚12531.84228515625 disk.drbd2165.in‚0 disk.sda.in‚0$ disk./.usageGB‚546.0989990234375 disk.rbd3.out‚0 disk.rbd2.out‚0& disk.nvme0n1.out‚12531.84228515625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.817307472229004 network.sent‚ 5.853782e+06 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 98.8046875 disk.sda3.out‚ 98.8046875# proc.memory.rssMB‚ 2755.34765625 disk.nvme0n1.in‚ 182.87890625 cpu‚26.068528161242774 ~ö¯2ü- _wandbruntime‚3208  _runtime‚3208ÈìÛÈý trainer/global_step‚3749 $ _timestamp‚1.784276016729872e+09 _runtime‚3208.855439254 _step‚305 ' lr-AdamW/pg1‚0.00043942824042567314 ' lr-AdamW/pg2‚0.00043942824042567314—8ÿ ÍþÇ _step‚305 ' lr-AdamW/pg1‚0.00043942824042567314 ' lr-AdamW/pg2‚0.00043942824042567314 trainer/global_step‚3749 $ _timestamp‚1.784276016729872e+09 _runtime‚3208.855439254±T oå2ÿ- _wandbruntime‚3208  _runtime‚3208:„ñpß'€ÙO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚306 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8056479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129631140184064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚569555801956352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34713613 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3962693817856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129631247373312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4006215117824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4006126540800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3962693817856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962352478720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29709854 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566628272263168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573518137395712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784276016730888e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14628790 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚569555784916992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚570584917498841 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29709856 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573518495774208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6572311 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34713613 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14628537 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573518495774208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34723590 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚569555801956352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29709856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34723590 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚570584559909591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3956304685783 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573518137395712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3962352478720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125625013643264 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573518137395712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3962352478720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125625013643264 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚129631247373312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4006215117824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4006126540800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3962693817856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962352478720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29709854 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566628272263168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573518137395712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784276016730888e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14628790 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚569555784916992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚570584917498841 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29709856 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573518495774208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6572311 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34713613 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14628537 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573518495774208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34723590²+/É2‚- _wandbruntime‚3209  _runtime‚3209»“¬Œê&ƒäM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125658442401792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573666391052288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129665699176960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440151 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29717486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34732656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚569703042985984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8058286 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚570732081635185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4007147495936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7842760173432453e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569703025946624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚569703025946624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14632339 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125658461014016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440151 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573666749437952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29717486 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3963706451968 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8058542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6574053 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963365105664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129665589897728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4007238162944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566774765895680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34732656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚3749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34722665 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3957315739505 _step‚307 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569703042985984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154FŒahb„éM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125658442401792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573666391052288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129665699176960 N>DY·èv#eviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440151 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29717486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34732656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚569703042985984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8058286 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚570732081635185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4007147495936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7842760173432453e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚569703025946624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚569703025946624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14632339 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125658461014016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440151 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573666749437952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29717486 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3963706451968 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8058542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6574053 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963365105664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129665589897728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4007238162944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566774765895680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34732656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚3749 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34722665 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3957315739505 _step‚307 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚569703042985984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P†‚ 2loss/SRqqt2/assignment_loss‚0.5754354253046282 4loss/FBt2/assignment_loss‚5.8134662629917376e-05 /loss/FRt2/detection_loss‚1.2547142322948186 1loss/SRqqt2/detection_loss‚0.6486487085925359 2loss/SRqqt1/assignment_loss‚0.6457165795178027 /loss/FRt1/detection_loss‚0.5636400849891098 _runtime‚3209.471402586 _step‚308 1loss/SRqqt1/detection_loss‚1.0029699579266107 /loss/FBt1/detection_loss‚0.5224530398313488 /loss/FBt2/detection_loss‚0.8470659651164267 %loss/total_loss‚8.997916468363453 epoch‚9 trainer/global_step‚3749 /loss/FRt1/assignment_loss‚1.816975969653409 /loss/FRt2/assignment_loss‚1.101607311142428 1loss/FBt1/assignment_loss‚0.01863105933170363 % _timestamp‚1.7842760173456552e+09Ïˉž ‡‡ /loss/FRt1/assignment_loss‚1.816975969653409 /loss/FRt2/assignment_loss‚1.101607311142428 1loss/FBt1/assignment_loss‚0.01863105933170363 % _timestamp‚1.7842760173456552e+09 2loss/SRqqt2/assignment_loss‚0.5754354253046282 4loss/FBt2/assignment_loss‚5.8134662629917376e-05 /loss/FRt2/detection_loss‚1.2547142322948186 1loss/SRqqt2/detection_loss‚0.6486487085925359 2loss/SRqqt1/assignment_loss‚0.6457165795178027 /loss/FRt1/detection_loss‚0.5636400849891098 _runtime‚3209.471402586 _step‚308 1loss/SRqqt1/detection_loss‚1.0029699579266107 /loss/FBt1/detection_loss‚0.5224530398313488 /loss/FBt2/detection_loss‚0.8470659651164267 %loss/total_loss‚8.997916468363453 epoch‚9 trainer/global_step‚3749´ê,ˆ‚(:’ ±ÈçÒו±¤ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖLäý‰‚(:ˆ ±ÈçÒ¡‚•§ disk.rbd3.in‚0 network.sent‚ 5.869846e+06 network.recv‚865130" disk.nvme0n1.out‚ 12541.2265625 disk.sda1.in‚0( proc.memory.availableMB‚ 529.59765625# proc.memory.rssMB‚ 2755.34765625* disk./.usagePercent‚36.649177136487324 disk.rbd0.in‚0 disk.sda.out‚ 99.12890625 disk.dm-1.in‚0 cpu‚26.06690295398932 disk.drbd2165.out‚0 disk.dm-0.out‚ 12541.2265625$ memory_percent‚96.76759243011475$ disk./.usageGB‚546.0992240905762 disk.dm-0.in‚ 183.59375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 99.12890625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 183.59375 disk.sda.in‚0 disk.sda3.out‚ 99.12890625 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚91.62255859375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 183.59375 disk.sda1.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚16.817307472229004 disk.rbd2.out‚0$ disk.nvme0c0n1.out‚ 12541.2265625 disk.rbd3.out‚0DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580930884379136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576917853437952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580931242757632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4057239879680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35168065 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35178042 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚580931242757632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577960709990099 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚310 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259519 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131353802832896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚3248.562024213 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35168065 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6659558 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚573953788817408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4007261722581 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4006921172691 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576917836398592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6659561 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8160336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013389319680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573953805856768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30091454 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4057332651008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131353691449344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚580930884379136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127296470181888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576917836398592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4013047980544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2sä·Ï“ßO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576917836398592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4013047980544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7842760564359734e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4013047980544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269498 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580930884379136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576917853437952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580931242757632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4057239879680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35168065 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35178042 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚580931242757632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577960709990099 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚310 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259519 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131353802832896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚3248.562024213 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚Ù笒  553967983 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35168065 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6659558 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚573953788817408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4007261722581 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4006921172691 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576917836398592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6659561 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8160336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013389319680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573953805856768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30091454 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4057332651008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131353691449344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚580930884379136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127296470181888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0¶ð7²2”- _wandbruntime‚3249  _runtime‚3249Jg§Ÿé&•ãM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚577065094467584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚574100299489280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4008272776415 $ _timestamp‚1.784276057047426e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286196 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30099088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35187108 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚581079496421376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚577065094467584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581079496421376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚311 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚577065077428224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6661303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127329880328192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578108589305055 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚574100316528640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4058356611584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578108231715693 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65276203 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚3249.173344099 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4014060607488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30099086 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014401953792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14823643 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4058265944576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚577065077428224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8162340 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286196 trainer/global_step‚3799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581079138035712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30099086 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30099088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127329898940416 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35177117 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4014401953792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8162596 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276203ÇšaÕî&–èM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4008272776415 $ _timestamp‚1.784276057047426e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286196 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30099088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35187108 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚581079496421376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚577065094467584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581079496421376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚311 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚577065077428224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6661303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127329880328192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578108589305055 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚574100316528640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4058356611584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578108231715693 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65276203 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚3249.173344099 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4014060607488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30099086 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014401953792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14823643 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4058265944576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚577065077428224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8162340 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286196 trainer/global_step‚3799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581079138035712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30099086 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30099088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127329898940416 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35177117 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4014401953792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8162596 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276203 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚577065094467584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚574100299489280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35622517 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6746808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35632494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585337217659857 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095571 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviÛÇòëøceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30473056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584279887880192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4064084821504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588343631362560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133076427883520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15011819 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584279904919552 _step‚314 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588343631362560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15011540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66105550 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269gÖÛvä'¥ÞO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15011819 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584279904919552 _step‚314 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588343631362560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15011540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66105550 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4064084821504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚584279887880192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚585336860070607 % _timestamp‚1.7842760963549724e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30473054 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128967908108288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4108433295360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4063743482368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚ 3288.48154827 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35622517 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6746808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35632494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585337217659857 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095571 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30473056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584279887880192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4064084821504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588343631362560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133076427883520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976ºaˆùž2¦- _wandbruntime‚3289  _runtime‚3289//M'ê&§äM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588492243404800 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8266828 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842760969672194e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065097455616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8267112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133110878083584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4058548713321 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4065097455616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15015381 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6748553 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581425833082880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚315 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4058889263323 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35641560 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584427145949184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30480686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584427145949184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112255 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4109541216768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35631569 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4064756109312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚581425850122240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3849 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4109444258304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35631569 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15015668 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269æ¶ÿ#å¨éM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35631569 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15015668 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588492243404800 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8266828 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842760969672194e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065097455616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8267112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133110878083584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4058548713321 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4065097455616 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15015381 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6748553 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581425833082880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚315 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4058889263323 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35641560 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚584427145949184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30480686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584427145949184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMºi0 onitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112255 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4109541216768 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35631569 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4064756109312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚581425850122240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3849 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4109444258304»„$t%2©- _wandbruntime‚3289  _runtime‚3289_|]Ôªþ 0loss/FRt1/assignment_loss‚1.9078434716498072 /loss/FRt2/detection_loss‚1.3585146234395145 1loss/SRqqt2/detection_loss‚0.6866365790451366 %loss/total_loss‚9.335690749873265 % _timestamp‚1.7842760969698586e+09 _runtime‚3289.095301777 _step‚316 2loss/SRqqt2/assignment_loss‚0.6018585744902873 0loss/FBt1/assignment_loss‚0.0525535562343943 /loss/FBt2/detection_loss‚0.7533596794653151 trainer/global_step‚3849 /loss/FRt2/assignment_loss‚1.068588575731546 2loss/SRqqt1/assignment_loss‚0.6722446927900597 1loss/SRqqt1/detection_loss‚1.0511027731346936 .loss/FBt1/detection_loss‚0.520203734262158 1loss/FBt2/assignment_loss‚0.07630100790593783 /loss/FRt1/detection_loss‚0.5864834817244149 epoch‚9Z¹ «ƒ 1loss/SRqqt2/detection_loss‚0.6866365790451366 %loss/total_loss‚9.335690749873265 % _timestamp‚1.7842760969698586e+09 _runtime‚3289.095301777 _step‚316 2loss/SRqqt2/assignment_loss‚0.6018585744902873 0loss/FBt1/assignment_loss‚0.0525535562343943 /loss/FBt2/detection_loss‚0.7533596794653151 trainer/global_step‚3849 /loss/FRt2/assignment_loss‚1.068588575731546 2loss/SRqqt1/assignment_loss‚0.6722446927900597 1loss/SRqqt1/detection_loss‚1.0511027731346936 .loss/FBt1/detection_loss‚0.520203734262158 1loss/FBt2/assignment_loss‚0.07630100790593783 /loss/FRt1/detection_loss‚0.5864834817244149 epoch‚9 0loss/FRt1/assignment_loss‚1.9078434716498072 /loss/FRt2/detection_loss‚1.3585146234395145¼Mÿ]“ž¬‚(:“ ‹ÉçÒÝû½¤ gpu.0.gpu‚86.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.258' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.251599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â^é`'­‚(:œ ‹ÉçÒ„Ш¦( disk.nvme0c0n1.out‚12587.33349609375 network.recv‚887253 disk.sda3.out‚ 101.00390625 disk.dm-0.in‚ 186.6796875 disk.sda1.in‚0 disk.rbd0.in‚0 network.sent‚ 6.002372e+06& disk.nvme0n1.out‚12587.33349609375 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.817307472229004$ memory_percent‚96.77221775054932( proc.memory.availableMB‚ 528.83984375$ disk./.usageGB‚546.0998954772949 disk.dm-1.out‚ 101.00390625 disk.rbd2.out‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 186.6796875 disk.sda2.in‚0# proc.memory.rssMB‚ 2755.34765625 disk.sda.out‚ 101.00390625 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚91.90380859375 disk.rbd3.in‚0 cpu‚26.05755878374297 proc.cpu.threads‚17) disk./.usagePercent‚36.64922219381335# disk.dm-0.out‚12587.33349609375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 186.6796875 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0GÍñ®‚(:„ šÉçÒ÷’š¥ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SKq"¯‚(:— šÉçÒþÔɧ network.sent‚ 6.065564e+06 disk.dm-0.in‚ 187.5078125 disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.77190780639648* proc.memory.percent‚16.817307472229004 cpu‚26.05609108793974 disk.rbd3.out‚0 disk.rbd2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875# disk.dm-0.out‚12594.60302734375 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 101.2109375& proc.memory.availableMB‚ 528.890625 network.recv‚891990$ disk./.usageGB‚546.1000099182129 disk.rbd1.out‚91.96630859375 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 101.2109375& disk.nvme0n1.out‚12594.60302734375 disk.rbd2.in‚0 disk.sda3.out‚ 101.2109375 proc.cpu.threads‚17 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 187.5078125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2755.34765625 disk.sda2.out‚0) disk./.usagePercent‚36.64922987403938 disk.rbd3.in‚0 disk.nvme0n1.in‚ 187.5078125( disk.nvme0c0n1.out‚12594.60302734375 disk.sda2.in‚0R_'¬2°- _wandbruntime‚3328  _runtime‚3328ú۷ɱà ' lr-AdamW/pg1‚0.00043646406571159644 ' lr-AdamW/pg2‚0.00043646406571159644 trainer/global_step‚3899 % _timestamp‚1.7842761362746298e+09 _runtime‚3328.400159322 _step‚317ԚةÎ²È ' lr-AdamW/pg1‚0.00043646406571159644 ' lr-AdamW/pg2‚0.00043646406571159644 trainer/global_step‚3899 % _timestamp‚1.7842761362746298e+09 _runtime‚3328.400159322 _step‚317½&(¾æ2³- _wandbruntime‚3328  _runtime‚3328£,æªà'´ÚO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚591641939361792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3899 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4114438984192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595756378345984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚591641956401152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66941602 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15204097 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚592713010151115 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588604856004608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8369782 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30854654 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚591641956401152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚595756736724480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588604873043968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚592713367740365 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134798957661696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4108494696397 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚318 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4114438984192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6834058 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130639346034688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114780323328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36076969 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15203840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚591641939361792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30854656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4114780323328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36086946 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚3328.401655707 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚595756736724480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6834061 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30854656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30854654 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8370036 % _timestamp‚1.7842761362758665e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4159712524800—ntMå'µßO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3899 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4114438984192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595756378345984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚591641956401152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66941602 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15204097 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚592713010151115 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588604856004608 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8369782 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30854654 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚591641956401152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚595756736724480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588604873043968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚592713367740365 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134798957661696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4108494696397 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚318 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4114438984192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6834058 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130639346034688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114780323328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36076969 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15203840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚591641939361792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30854656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4114780323328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36086946 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚3328.401655707 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚595756736724480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6834061 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30854656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30854654 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8370036 % _timestamp‚1.7842761362758665e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4159712524800 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚591641939361792¾jºu+2¶- _wandbruntime‚3329  _runtime‚3329/`qß·ãM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚595904632002560 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚595904990388224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588751383715840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66958300 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4115792957440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8371802 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚592860531876709 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚591789197430784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚591789197430784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚591789180391424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UIDeO~6 viceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8372070 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30862288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚595904990388224 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15207605 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4115451611136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4160644847616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134833510307840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588751366676480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130672774793216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30862286 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4160735514624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592860889466071 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36096012 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4109165200229 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30862286 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4109505750231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚3899 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6835806 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36096012 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130672756180992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4115451611136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948307 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4115792957440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4160735514624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592860889466071 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36096012 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4109165200229 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30862286 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4109505750231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚3899 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6835806 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36096012 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130672756180992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4115451611136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948307 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4115792957440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4115792957440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8371802 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚592860531876709 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚591789197430784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚591789197430784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚591789180391424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8372070 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30862288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚595904990388224 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15207605 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4115451611136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4160644847616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134833510307840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588751366676480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130672774793216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30862286 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¿ø2¹- _wandbruntime‚3329  _runtime‚3329„Vúµº /loss/FRt1/detection_loss‚0.5314290586748871 1loss/SRqqt2/detection_loss‚0.6954322997873607 /loss/FBt2/detection_loss‚0.6366529823431084 %loss/total_loss‚8.951876887276413 epoch‚9 trainer/global_step‚3899 % _timestamp‚1.7842761368980927e+09 _step‚320 0loss/FRt2/assignment_loss‚1.0637302291782007 2loss/SRqqt1/assignment_loss‚0.5104219133298831 1loss/FBt2/assignment_loss‚0.04262904659252113 0loss/FRt1/assignment_loss‚1.9460540889411844 2loss/SRqqt2/assignment_loss‚0.5861500717033538 /loss/FBt1/detection_loss‚0.5750578372568196 1loss/FBt1/assignment_loss‚0.05065886877602324 /loss/FRt2/detection_loss‚1.2946624973513847 1loss/SRqqt1/detection_loss‚1.0189979933416857 _runtime‚3329.023538769Òl׊ »† 2loss/SRqqt1/assignment_loss‚0.5104219133298831 1loss/FBt2/assignment_loss‚0.04262904659252113 0loss/FRt1/assignment_loss‚1.9460540889411844 2loss/SRqqt2/assignment_loss‚0.5861500717033538 /loss/FBt1/detection_loss‚0.5750578372568196 1loss/FBt1/assignment_loss‚0.05065886877602324 /loss/FRt2/detection_loss‚1.2946624973513847 1loss/SRqqt1/detection_loss‚1.0189979933416857 _runtime‚3329.023538769 /loss/FRt1/detection_loss‚0.5314290586748871 1loss/SRqqt2/detection_loss‚0.6954322997873607 /loss/FBt2/detection_loss‚0.6366529823431084 %loss/total_loss‚8.951876887276413 epoch‚9 trainer/global_step‚3899 % _timestamp‚1.7842761368980927e+09 _step‚320 0loss/FRt2/assignment_loss‚1.0637302291782007ÀÈX*Ü‘¼‚(:† ©ÉçÒ½ùä¥ gpu.0.gpu‚9.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šû³#½‚(:˜ ©ÉçÒ²¸ý¨ disk.dm-0.in‚ 188.22265625' disk.nvme0c0n1.out‚12605.1201171875 network.sent‚ 6.083715e+06 disk.rbd3.out‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 188.22265625 disk.sda.in‚0 cpu‚26.054560033559547 disk.rbd1.in‚ 0.01171875 disk.drbd2165.in‚0) disk./.usagePercent‚36.64922603392636 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 network.recv‚895541 disk.sda3.out‚ 101.6328125 disk.rbd1.out‚91.96630859375" disk.dm-0.out‚12605.1201171875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 577.83203125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚12605.1201171875 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 188.22265625 disk.sda.out‚ 101.6328125# proc.memory.rssMB‚ 2639.26953125( proc.memory.percent‚16.1088228225708 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚96.47319316864014$ disk./.usageGB‚546.0999526977539 disk.dm-1.out‚ 101.6328125;eYŒž¾‚(:“ ¸ÉçÒæË࣠gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©™Î¿‚(:Œ ¸ÉçÒ¨ßê¥ disk.dm-1.in‚0 disk.rbd1.out‚92.04443359375 disk.drbd2165.out‚0 disk.rbd2.out‚0% disk.nvme0n1.out‚12613.0947265625" disk.nvme0c0n1.in‚ 189.05078125 disk.sda3.out‚101.875# memory_percent‚96.4728832244873) disk./.usagePercent‚36.64922142579074 network.sent‚ 6.135821e+06 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 577.8828125 disk.nvme0n1.in‚ 189.05078125 disk.rbd0.out‚0 disk.sda.out‚101.875 cpu‚26.053115741585625$ disk./.usageGB‚546.0998840332031" disk.dm-0.out‚12613.0947265625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚12613.0947265625 disk.sda2.out‚0 disk.dm-1.out‚101.875 disk.rbd1.in‚ 0.01171875* proc.memory.percent‚16.109013557434082 disk.dm-0.in‚ 189.05078125 network.recv‚900094 disk.sda2.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2639.30078125o¥­“À‚(:ˆ ÇÉçÒÁ˜¡¤ gpu.0.gpu‚88.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚255.341' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÌÇá&Á‚(:› ÇÉçÒº®®¦ network.recv‚903724 disk.rbd3.in‚0! disk.dm-0.out‚12619.626953125 disk.nvme0n1.in‚ 189.05078125( proc.memory.availableMB‚ 577.94140625 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0$ memory_percent‚96.47252559661865 network.sent‚ 6.138743e+06$ disk./.usageGB‚546.0951232910156 disk.rbd2.out‚0 disk.sda.out‚ 102.06640625 proc.cpu.threads‚17$ disk.nvme0n1.out‚12619.626953125& disk.nvme0c0n1.out‚12619.626953125# proc.memory.rssMB‚ 2639.30078125 cpu‚26.051698760507705 disk.rbd1.out‚92.04443359375 disk.dm-0.in‚ 189.05078125 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0* disk./.usagePercent‚36.648901928388014 disk.rbd2.in‚0 disk.sda3.in‚0* proc.memory.percent‚16.109013557434082 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 189.05078125 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 102.06640625 disk.sda3.out‚ 102.06640625{ß®b2Â- _wandbruntime‚3368  _runtime‚3368Dã—ÉÃà ' lr-AdamW/pg1‚0.00043545021408064334 ' lr-AdamW/pg2‚0.00043545021408064334 trainer/global_step‚3949 % _timestamp‚1.7842761761394897e+09 _runtime‚3368.265073734 _step‚321SËÅËÎÄÈ _step‚321 ' lr-AdamW/pg1‚0.00043545021408064334 ' lr-AdamW/pg2‚0.00043545021408064334 trainer/global_step‚3949 % _timestamp‚1.7842761761394897e+09 _runtime‚3368.265073734Á€Äx2Å- _wandbruntime‚3368  _runtime‚3368õ¯[Ì6!ÆÚO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31236254 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15395658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3368.266354953 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚603169483707904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31236256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8474350 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132310783961088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚595930406637568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36541398 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6921311 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31236254 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚599004007882752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600089160231623 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4158770633415 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4159111183305 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67767675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600089517820873 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595930389598208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚322 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599003990843392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842761761405597e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚603169125329408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4165475825152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4165134486016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15395938 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4210848009728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚599004007882752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 trainer/global_step‚3949 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603169125329408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4165475825152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888]ð3Ýå'ÇßO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132310765348864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136521728936448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599003990843392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8474627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67777654 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36531421 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603169125329408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4165475825152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31236254 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15395658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3368.266354953 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚603169483707904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31236256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8474350 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132310783961088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚595930406637568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36541398 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6921311 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31236254 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚599004007882752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600089160231623 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4158770633415 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4159111183305 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67767675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600089517820873 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595930389598208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚322 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599003990843392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842761761405597e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚603169125329408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4165475825152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4165134486016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15395938 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4210848009728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚599004007882752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 trainer/global_step‚3949 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36540473 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599151231873024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6923053 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36550464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚596076917309440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚603317378985984 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚323 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚3368.879948555 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8476426 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4211959175168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4166147112960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67784359 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31243886 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67784359 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4166488459264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31243886 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6923056 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4166488459264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4160122237139 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31243888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15399479 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31243888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚596076900270080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144Ê£°Mí&ÊçM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36540473 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599151231873024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6923053 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36550464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚596076917309440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚603317378985984 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚323 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚3368.879948555 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8476426 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4211959175168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4166147112960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67784359 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31243886 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67784359 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4166488459264 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31243886 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6923056 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4166488459264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4160122237139 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31243888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15399479 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31243888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚596076900270080ÃvL2Ë- _wandbruntime‚3368  _runtime‚3368bRàÌ€ epoch‚9 _runtime‚ 3368.8812494 2loss/FBt2/assignment_loss‚0.014377820337634175 /loss/FRt1/detection_loss‚0.5445173644471808 /loss/FBt2/detection_loss‚0.7852608056908305 % _timestamp‚1.7842761767559133e+09 _step‚324 0loss/FRt1/assignment_loss‚1.9443497162352326 2loss/FBt1/assignment_loss‚0.024546543672273494 0loss/SRqqt1/detection_loss‚1.139201089284579 0loss/FRt2/assignment_loss‚1.0252955044850265 2loss/SRqqt1/assignment_loss‚0.7276678947719324 /loss/FRt2/detection_loss‚1.4070494279975423 trainer/global_step‚3949 1loss/SRqqt2/assignment_loss‚0.586349148938366 1loss/SRqqt2/detection_loss‚0.6555477854124866 0loss/FBt1/detection_loss‚0.45599881588564944 %loss/total_loss‚9.310161917158734ˆ×¹¾ Í… 1loss/SRqqt2/assignment_loss‚0.586349148938366 1loss/SRqqt2/detection_loss‚0.6555477854124866 0loss/FBt1/detection_loss‚0.45599881588564944 %loss/total_loss‚9.310161917158734 epoch‚9 _runtime‚ 3368.8812494 2loss/FBt2/assignment_loss‚0.014377820337634175 /loss/FRt1/detection_loss‚0.5445173644471808 /loss/FBt2/detection_loss‚0.7852608056908305 % _timestamp‚1.7842761767559133e+09 _step‚324 0loss/FRt1/assignment_loss‚1.9443497162352326 2loss/FBt1/assignment_loss‚0.024546543672273494 0loss/SRqqt1/detection_loss‚1.139201089284579 0loss/FRt2/assignment_loss‚1.0252955044850265 2loss/SRqqt1/assignment_loss‚0.7276678947719324 /loss/FRt2/detection_loss‚1.4070494279975423 trainer/global_step‚3949Äæt–ü“΂(:ˆ ÖÉçÒÛöí£ gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ÐÏwÏ‚(:‘ ÖÉçÒö±¥ disk.sda3.in‚0 disk.dm-1.out‚ 102.45703125 disk.rbd3.out‚0 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 102.45703125$ memory_percent‚96.47552967071533 network.recv‚907122 disk.sda1.in‚0 disk.rbd1.out‚92.10693359375 disk.dm-0.in‚ 189.765625 proc.cpu.threads‚17) disk./.usagePercent‚36.64896516224898 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 102.45703125% disk.nvme0c0n1.out‚12627.30078125 disk.dm-1.in‚0* proc.memory.percent‚16.109013557434082$ disk./.usageGB‚546.0960655212402 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 189.765625 network.sent‚ 6.143832e+06 disk.rbd0.out‚0# proc.memory.rssMB‚ 2639.30078125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.out‚12627.30078125# disk.nvme0n1.out‚12627.30078125( proc.memory.availableMB‚ 577.44921875 cpu‚26.050245495011588 disk.nvme0c0n1.in‚ 189.765625ÙÉT¬žÐ‚(:“ åÉçÒ¥ gpu.0.gpu‚58.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.398' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.479600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!. Ñ‚(: åÉçÒçóÚ§ network.sent‚ 6.206813e+06 disk.sda3.out‚ 102.6953125 disk.sda1.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2639.3046875 cpu‚26.048824688213042 disk.rbd3.out‚0 disk.rbd0.in‚0 network.recv‚911781 disk.rbd1.out‚92.10693359375 disk.rbd3.in‚0 disk.sda.out‚ 102.6953125( proc.memory.availableMB‚ 577.48828125* proc.memory.percent‚16.109037399291992) disk./.usagePercent‚36.64891984891541$ disk./.usageGB‚546.0953903198242 disk.sda2.out‚0 disk.dm-1.out‚ 102.6953125 disk.nvme0n1.in‚ 190.59375% disk.nvme0n1.out‚12637.6962890625 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 190.59375" disk.dm-0.out‚12637.6962890625 disk.rbd2.in‚0' disk.nvme0c0n1.out‚12637.6962890625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 190.59375 disk.sda.in‚0 disk.rbd2.out‚0$ memory_percent‚96.47529125213623 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17@•Ò‚(:’ ôÉçÒÃÆܦ gpu.0.gpu‚91.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚280.405' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.08099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 m>#!Ó‚(:– ôÉçÒœý㪠network.sent‚ 6.214137e+06 disk.nvme0n1.in‚ 191.30859375 disk.sda.out‚ 102.96875 disk.dm-1.out‚ 102.96875 disk.sda1.out‚0( disk.nvme0c0n1.out‚12644.58251953125 disk.sda3.in‚0 disk.dm-1.in‚0* proc.memory.percent‚16.109037399291992) disk./.usagePercent‚36.64895543396267$ disk./.usageGB‚546.0959205627441 disk.sda2.out‚0 disk.sda3.out‚ 102.96875" proc.memory.rssMB‚ 2639.3046875 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875( proc.memory.availableMB‚ 577.48046875 proc.cpu.threads‚17 network.recv‚915211 disk.rbd3.in‚0# disk.dm-0.out‚12644.58251953125 disk.rbd1.out‚92.18505859375$ memory_percent‚96.47533893585205 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 191.30859375 disk.sda2.in‚0 cpu‚26.047404652430668 disk.rbd3.out‚0 disk.dm-0.in‚ 191.30859375 disk.rbd2.out‚0& disk.nvme0n1.out‚12644.58251953125 disk.sda1.in‚0 disk.rbd0.in‚0kgÒ¯2Ô- _wandbruntime‚3408  _runtime‚3408Ü×Ê·ÉÕà ' lr-AdamW/pg1‚0.00043442357359420894 ' lr-AdamW/pg2‚0.00043442357359420894 trainer/global_step‚3999 % _timestamp‚1.7842762161411574e+09 _runtime‚3408.266731287 _step‚325dg`BÎÖÈ _step‚325 ' lr-AdamW/pg1‚0.00043442357359420894 ' lr-AdamW/pg2‚0.00043442357359420894 trainer/global_step‚3999 % _timestamp‚1.7842762161411574e+09 _runtime‚3408.266731287ÅKå2×- _wandbruntime‚3408  _runtime‚3408ŽÃzà'ØÚO KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31617856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610581872312832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842762161421888e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31617854 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚606366042324992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4216171326976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚606366059364352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610581872312832 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610582230691328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4215829987840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4216171326976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36985873 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4262195248128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607465667901381 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚610582230691328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613706 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15587962 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603727 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8579401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4262281728000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36995850 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31617856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36995850 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31617854 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚606366059364352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚3999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138244398523392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607465310312131 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4209727670213×üžå'ÙßO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚606366042324992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133982203275264 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚603255923191808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138244503615488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36985873 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7008558 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610582230691328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4215829987840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4216171326976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36985873 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4262195248128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607465667901381 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚610582230691328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613706 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15587962 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603727 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8579401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4262281728000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36995850 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31617856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36995850 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31617854 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚606366059364352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚3999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138244398523392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607465310312131 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4209727670213 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31617856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610581872312832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842762161421888e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31617854 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚606366042324992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4216171326976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚606366059364352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610581872312832 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784276216755782e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚606513283354624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚3408.881637063 _step‚327 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216842614784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138278956707328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603402450903040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4210398174045 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15591812 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4263213297152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36994925 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15591552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8581249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37004916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7010303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138278845330944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630404 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/activeí(å0u _bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620411 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31625486 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134015632033792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630404 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚610730484355072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31625488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4216842614784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0-ºæ?î&ÜèM BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620411 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31625486 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134015632033792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630404 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚610730484355072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31625488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4216842614784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784276216755782e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚606513283354624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚3408.881637063 _step‚327 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216842614784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138278956707328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603402450903040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4210398174045 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15591812 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4263213297152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36994925 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15591552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8581249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37004916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7010303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138278845330944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630404 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.020166363567113876 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7823382616043091 *jet/accuracy_2_of_4‚0.7970789074897766 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951270818710327 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9808148741722107 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07064647227525711 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20585858821868896 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8224549889564514 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 _runtime‚3425.972666062 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02196200005710125 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7807216644287109 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8214024901390076 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8674906492233276 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.653199315071106 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47926658391952515 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.6293362379074097 ) Purity/0FRt1SRqqt2FBt/FBt_purity‚0.75 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21841105818748474 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960944056510925 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5429166555404663 particle/accuracy_5_of_5‚0 % _timestamp‚1.7842762338464508e+09 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3158605992794037 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.266049325466156 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7877885699272156 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9033137559890747 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6808703541755676 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7859028577804565 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35648149251937866 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 *jet/accuracy_1_of_3‚0.9249552488327026 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 _step‚329 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9734878540039062 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20887577533721924 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7166478037834167 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021235177293419838 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6945794224739075 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9917820692062378 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25566765666007996 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7369016408920288 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.207503542304039 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02074894867837429 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5850262641906738 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7258180975914001 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.7004305124282837 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6713519096374512 particle/accuracy_1_of_1‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.7024948000907898 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8196523785591125 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4791666567325592 *jet/accuracy_1_of_4‚0.9815526604652405 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.028587184846401215 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.8050734400749207 particle/accuracy_3_of_3‚1 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6674185991287231 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9904074668884277 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8147770762443542 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_4‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7845920324325562 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1525000035762787 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22907912731170654 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5 jet/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.41868236660957336 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941943883895874 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6231105327606201 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7834845781326294 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7873609662055969 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8785164952278137 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09168989211320877 *"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1261482685804367 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7610194683074951 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6998934745788574 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.71022629737854 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18929898738861084 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7381259202957153 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3564665913581848 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48696309328079224 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05662251263856888 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960944056510925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.41600650548934937 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3828125 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5085291266441345 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.266912043094635 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02227577194571495 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8674906492233276 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (loss/val_total_loss‚8.74324739087622 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26752734184265137 'particle/f_score‚0.6417241096496582 +jet/accuracy_3_of_4‚0.08654024451971054 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7537034153938293 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2199534922838211 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02074894867837429 *jet/accuracy_2_of_3‚0.5900412797927856 trainer/global_step‚3999 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.740386426448822 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.923279881477356 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8240740895271301 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7609344720840454 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7008957862854004 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19555509090423584 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31345677375793457 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992900550365448 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22753913700580597 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22893479466438293 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5149844288825989 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03752295300364494 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9230749011039734 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.783170223236084 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6915144920349121 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2055010348558426 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4822397828102112 .particle/accuracy_3_of_4‚0.997619092464447 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2463291883468628 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9808259010314941 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 .particle/accuracy_4_of_4‚0.934293270111084 6validation_average_jet_accuracy‚0.3953087329864502 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25365564227104187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7393649220466614 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7182793617248535 particle/accuracy_1_of_2‚1 epoch‚9 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022497428581118584 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.7004305124282837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7955405116081238 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971181154251099 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25566765666007996 +particle/specificity‚0.9092098474502563 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35099706053733826 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.807331919670105 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.786769688129425 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5005739331245422 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965853095054626 +jet/accuracy_2_of_2‚0.16656403243541718 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991644024848938 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.7017605900764465 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6649914383888245 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2247275412082672 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 (particle/accuracy‚0.8491056561470032 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951848387718201 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9750511646270752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10925813764333725 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31414803862571716 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5777142643928528 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10811961442232132 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20591680705547333 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9911824464797974 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895260334014893 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5¨ þ¶Ÿäüz 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25365564227104187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7393649220466614 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7182793617248535 particle/accuracy_1_of_2‚1 epoch‚9 ) Purity/*FRt1SRqqt2FÌݸÿã;Bt/FRt_purity‚0.75 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022497428581118584 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.7004305124282837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7955405116081238 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971181154251099 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25566765666007996 +particle/specificity‚0.9092098474502563 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35099706053733826 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.807331919670105 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.786769688129425 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5005739331245422 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965853095054626 +jet/accuracy_2_of_2‚0.16656403243541718 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991644024848938 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.7017605900764465 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6649914383888245 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2247275412082672 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 (particle/accuracy‚0.8491056561470032 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951848387718201 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9750511646270752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10925813764333725 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31414803862571716 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5777142643928528 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10811961442232132 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20591680705547333 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9911824464797974 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895260334014893 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8629826903343201 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9768300652503967 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7694355845451355 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7430312633514404 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6996307373046875 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04536489024758339 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.053635772317647934 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24715478718280792 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.020166363567113876 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7823382616043091 *jet/accuracy_2_of_4‚0.7970789074897766 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951270818710327 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9808148741722107 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07064647227525711 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20585858821868896 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8224549889564514 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 _runtime‚3425.972666062 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02196200005710125 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7807216644287109 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8214024901390076 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8674906492233276 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.653199315071106 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47926658391952515 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.6293362379074097 ) Purity/0FRt1SRqqt2FBt/FBt_purity‚0.75 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21841105818748474 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960944056510925 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5429166555404663 particle/accuracy_5_of_5‚0 % _timestamp‚1.7842762338464508e+09 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3158605992794037 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.266049325466156 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7877885699272156 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9033137559890747 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6808703541755676 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7859028577804565 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35648149251937866 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 *jet/accuracy_1_of_3‚0.9249552488327026 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 _step‚329 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9734878540039062 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20887577533721924 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7166478037834167 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021235177293419838 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6945794224739075 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9917820692062378 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25566765666007996 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7369016408920288 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.207503542304039 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02074894867837429 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5850262641906738 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7258180975914001 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.7004305124282837 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6713519096374512 particle/accuracy_1_of_1‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.7024948000907898 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8196523785591125 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4791666567325592 *jet/accuracy_1_of_4‚0.9815526604652405 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.028587184846401215 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.8050734400749207 particle/accuracy_3_of_3‚1 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6674185991287231 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9904074668884277 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8147770762443542 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_4‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7845920324325562 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1525000035762787 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22907912731170654 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5 jet/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.41868236660957336 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941943883895874 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6231105327606201 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7834845781326294 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7873609662055969 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8785164952278137 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09168989211320877 *"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1261482685804367 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7610194683074951 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6998934745788574 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.71022629737854 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18929898738861084 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7381259202957153 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3564665913581848 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48696309328079224 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05662251263856888 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960944056510925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.41600650548934937 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3828125 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5085291266441345 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.266912043094635 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02227577194571495 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8674906492233276 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (loss/val_total_loss‚8.74324739087622 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26752734184265137 'particle/f_score‚0.6417241096496582 +jet/accuracy_3_of_4‚0.08654024451971054 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7537034153938293 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2199534922838211 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02074894867837429 *jet/accuracy_2_of_3‚0.5900412797927856 trainer/global_step‚3999 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.740386426448822 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.923279881477356 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8240740895271301 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7609344720840454 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7008957862854004 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19555509090423584 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31345677375793457 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992900550365448 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22753913700580597 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22893479466438293 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5149844288825989 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03752295300364494 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9230749011039734 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.783170223236084 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6915144920349121 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2055010348558426 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4822397828102112 .particle/accuracy_3_of_4‚0.997619092464447 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2463291883468628 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9808259010314941 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 .particle/accuracy_4_of_4‚0.934293270111084 6validation_average_jet_accuracy‚0.3953087329864502É r$“å‚(:ˆ ’ÊçÒ«ƒú¤ gpu.0.gpu‚47.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½= +*æ‚(:Ÿ ’ÊçÒ•õú§( disk.nvme0c0n1.out‚12903.31298828125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 103.70703125 disk.dm-0.in‚ 192.13671875( proc.memory.availableMB‚ 742.86328125* proc.memory.percent‚16.705942153930664* disk./.usagePercent‚36.648982058746235$ disk./.usageGB‚546.0963172912598 disk.rbd0.in‚0 disk.dm-1.out‚ 103.70703125 disk.rbd1.out‚92.31005859375 cpu‚26.06262694163196 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 6.293136e+06 disk.nvme0n1.in‚ 192.13671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd3.in‚0$ memory_percent‚95.46592235565186 disk.rbd2.in‚0& disk.nvme0n1.out‚12903.31298828125" disk.nvme0c0n1.in‚ 192.13671875 disk.sda.out‚ 103.70703125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚12903.31298828125 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.01171875" proc.memory.rssMB‚ 2737.1015625 network.recv‚922985>£éç‚(:… ¡ÊçÒç˪¥ gpu.0.gpu‚25.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚186.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FYÉä$è‚(:™ ¡ÊçÒÂçÙ§ disk.sda.out‚ 103.94140625 proc.cpu.threads‚17 network.sent‚ 6.339671e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 103.94140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 103.94140625 disk.rbd1.in‚ 0.01171875$ memory_percent‚95.46573162078857 cpu‚26.061146335424 disk.rbd2.out‚0& disk.nvme0n1.out‚13158.04931640625! disk.nvme0c0n1.in‚ 192.8515625 disk.sda2.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚13158.04931640625* proc.memory.percent‚16.705942153930664$ disk./.usageGB‚546.0921325683594 disk.rbd2.in‚0 disk.nvme0n1.in‚ 192.8515625 disk.sda1.in‚0 network.recv‚927103( proc.memory.availableMB‚ 742.89453125 disk.sda1.out‚0( disk.nvme0c0n1.out‚13158.04931640625 disk.sda.in‚0" proc.memory.rssMB‚ 2737.1015625) disk./.usagePercent‚36.64870121848117 disk.rbd1.out‚92.31005859375 disk.rbd3.out‚0 disk.dm-0.in‚ 192.8515625 disk.rbd0.out‚0DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619941919154688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616768200921203 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚612490137038848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612490119999488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37510652 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615657677496320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7098331 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7098328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚331 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619941919154688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8715426 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140001184779264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37520629 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4331631477760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541ÿ²-à´ðÞO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15814029 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32120122 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140001287774208 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630772 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32120122 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615657660456960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8715698 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4284600036864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616767843331953 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15813754 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚331 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_traz‹fÈ0in_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619941919154688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8715426 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140001184779264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37520629 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4331631477760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619941919154688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616768200921203 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚612490137038848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612490119999488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37510652 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615657677496320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7098331 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7098328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37519704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7100073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37519704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15817611 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8717538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4332664698368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32127754 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69647456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140035743454720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615804918525952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4285271324672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612636630671360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4285612670976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚616915722646909 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620090531196928 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7100076 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140035646758400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚4049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615804901486592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32127754 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135703000672256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚620090531196928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4285271324672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32127752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚620090172811264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32127752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279074936189 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 $ _timestamp‚1.784276275869762e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615804901486592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4278734386187 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37519704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7100073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37519704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15817611 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8717538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4332664698368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32127754 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69647456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140035743454720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615804918525952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4285271324672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612636630671360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4285612670976 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚616915722646909 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620090531196928 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7100076 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140035646758400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚4049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615804901486592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32127754 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135703000672256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚620090531196928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4285271324672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ì › 2ô- _wandbruntime‚3467  _runtime‚3467gïÔù õ† 3loss/SRqqt1/assignment_loss‚0.48040258320512125 2loss/FBt1/assignment_loss‚0.014881330427967576 0loss/SRqqt1/detection_loss‚0.994077722241497 %loss/total_loss‚8.811243105753546 0loss/FRt2/assignment_loss‚1.1240289206954017 4loss/FBt2/assignment_loss‚0.00019096466697402344 epoch‚10 % _timestamp‚1.7842762758723552e+09 _runtime‚3467.997697954 2loss/SRqqt2/assignment_loss‚0.5399755805527753 /loss/FRt1/detection_loss‚0.5335811187405402 .loss/FRt2/detection_loss‚1.308146655863021 trainer/global_step‚4049 _step‚333 0loss/FRt1/assignment_loss‚1.8710147795504075 1loss/SRqqt2/detection_loss‚0.6945112780306816 0loss/FBt1/detection_loss‚0.48035173589111463 /loss/FBt2/detection_loss‚0.7700804358880434™`Ciö‹ 1loss/SRqqt2/detection_loss‚0.6945112780306816 0loss/FBt1/detection_loss‚0.48035173589111463 /loss/FBt2/detection_loss‚0.7700804358880434 3loss/SRqqt1/assignment_loss‚0.48040258320512125 2loss/FBt1/assignment_loss‚0.014881330427967576 0loss/SRqqt1/detection_loss‚0.994077722241497 %loss/total_loss‚8.811243105753546 0loss/FRt2/assignment_loss‚1.1240289206954017 4loss/FBt2/assignment_loss‚0.00019096466697402344 epoch‚10 % _timestamp‚1.7842762758723552e+09 _runtime‚3467.997697954 2loss/SRqqt2/assignment_loss‚0.5399755805527753 /loss/FRt1/detection_loss‚0.5335811187405402 .loss/FRt2/detection_loss‚1.308146655863021 trainer/global_step‚4049 _step‚333 0loss/FRt1/assignment_loss‚1.8710147795504075͆¾†8÷‚(:’ ¿ÊçÒß噤 gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.402' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.28039999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í6µ"ø‚(:— ¿ÊçÒù’“¦ disk.rbd0.out‚0 disk.sda.out‚ 104.4453125 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚92.38818359375 disk.rbd2.out‚0 disk.nvme0n1.in‚ 194.39453125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0" memory_percent‚95.467209815979 cpu‚26.05816811451109 proc.cpu.threads‚17( proc.memory.availableMB‚ 742.65234375" proc.memory.rssMB‚ 2737.1015625* proc.memory.percent‚16.705942153930664 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚13173.2294921875 disk.dm-1.out‚ 104.4453125* disk./.usagePercent‚36.648784932944864" disk.nvme0c0n1.in‚ 194.39453125 disk.rbd2.in‚0 disk.dm-0.in‚ 194.39453125 network.recv‚934144% disk.nvme0n1.out‚13173.2294921875 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 6.347789e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0$ disk./.usageGB‚546.0933799743652 disk.sda3.out‚ 104.4453125" disk.dm-0.out‚13173.2294921875 disk.sda1.in‚0 disk.sda1.out‚0Æøñ¯“ù‚(:ˆ ÎÊçÒ…–•¤ gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚240.261' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.0522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÁØú‚(:Ž ÎÊçÒÖ Á¦ disk.dm-1.out‚ 104.73828125 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.out‚ 13180.46875 disk.nvme0n1.out‚ 13180.46875 network.recv‚938567 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.out‚ 13180.46875 network.sent‚ 6.412898e+06* disk./.usagePercent‚36.648693282247606 disk.sda2.out‚0$ disk./.usageGB‚546.0920143127441 disk.rbd0.in‚0 disk.rbd1.out‚92.45068359375 disk.rbd2.in‚0 disk.rbd2.out‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 195.22265625 disk.sda.out‚ 104.73828125 disk.sda1.in‚0$ memory_percent‚95.46751976013184' proc.memory.availableMB‚ 742.6015625* proc.memory.percent‚16.705965995788574 disk.sda1.out‚0 disk.rbd1.in‚ 0.01171875 disk.nvme0n1.in‚ 195.22265625 cpu‚26.056786425996957 disk.dm-0.in‚ 195.22265625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2737.10546875 disk.sda3.out‚ 104.73828125 disk.dm-1.in‚0r›2û- _wandbruntime‚3507  _runtime‚3507íÐì•‚üà % _timestamp‚1.7842763152505455e+09 _runtime‚3507.376112872 _step‚334 ' lr-AdamW/pg1‚0.00043233219062886524>š G ' lr-AdamW/pg2‚0.00043233219062886524 trainer/global_step‚4099ûTÎýÈ _runtime‚3507.376112872 _step‚334 ' lr-AdamW/pg1‚0.00043233219062886524 ' lr-AdamW/pg2‚0.00043233219062886524 trainer/global_step‚4099 % _timestamp‚1.7842763152505455e+09Îî0Û2þ- _wandbruntime‚3507  _runtime‚3507Ðnñà'ÿÚO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37975081 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4383010608640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37965104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623019711938560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137341009840128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16005837 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7185581 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚624144351001711 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚624143993412461 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4335295538688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476803 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7185578 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32501720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37975081 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627354666138112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚623019711938560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8820518 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70466824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141724001836544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137340991227904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627354666138112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4335295538688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8820259 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4383103379968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842763152517023e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4334954199552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4328680369263 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627355024516608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4334954199552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16006099 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552Ô¬Ýßå'€ßO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627354666138112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4335295538688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8820259 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4383103379968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842763152517023e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4334954199552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4328680369263 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚627355024516608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4334954199552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16006099 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37975081 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4383010608640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37965104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623019711938560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137341009840128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16005837 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7185581 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚624144351001711 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚624143993412461 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4335295538688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476803 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7185578 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32501720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37975081 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627354666138112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚623019711938560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8820518 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70466824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141724001836544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137340991227904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Ï wö2- _wandbruntime‚3507  _runtime‚3507Ðuò%ê&‚äM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4329691423097 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚623166970007552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493501 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4384122855424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4335966826496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4336308172800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7187323 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624291515138055 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4336308172800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4329350873095 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627502919794688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627503278180352 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4335966826496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8822332 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37984147 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚623166970007552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚624291872727417 _runtime‚3507.989649595 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4384036382720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627503278180352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141758561454080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚623166952968192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37974156 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619962164264960 % _timestamp‚1.7842763158637977e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483508 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16009655 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8822605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70493501 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493501 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4384122855424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4335966826496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4336308172800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7187323 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_ensßÙ•Od/requested_bytes.all.freed‚624291515138055 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4336308172800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4329350873095 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627502919794688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627503278180352 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4335966826496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8822332 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37984147 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚623166970007552 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚624291872727417 _runtime‚3507.989649595 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4384036382720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627503278180352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141758561454080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚623166952968192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37974156 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619962164264960 % _timestamp‚1.7842763158637977e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483508 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16009655 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8822605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70493501 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591Б„Ü)2„- _wandbruntime‚3507  _runtime‚3507RÐÌ…€ trainer/global_step‚4099 0loss/FRt1/assignment_loss‚1.8757030143685962 2loss/SRqqt1/assignment_loss‚0.4809711306992265 2loss/FBt2/assignment_loss‚0.007213530557473186 /loss/FRt1/detection_loss‚0.5447297199563348 /loss/FBt2/detection_loss‚0.8355356209256031 "loss/total_loss‚8.953465068442 $ _timestamp‚1.784276315866087e+09 0loss/FRt2/assignment_loss‚1.1604319787025452 2loss/SRqqt2/assignment_loss‚0.5421868913383916 1loss/SRqqt1/detection_loss‚0.9715563005321667 0loss/FBt1/detection_loss‚0.48771333010585677 epoch‚10 /loss/FRt2/detection_loss‚1.3140916009469046 _runtime‚3507.991431508 _step‚337 1loss/FBt1/assignment_loss‚0.03716814898921887 1loss/SRqqt2/detection_loss‚0.6961638013196825Mæ]' †… 0loss/FRt1/assignment_loss‚1.8757030143685962 2loss/SRqqt1/assignment_loss‚0.4809711306992265 2loss/FBt2/assignment_loss‚0.007213530557473186 /loss/FRt1/detection_loss‚0.5447297199563348 /loss/FBt2/detection_loss‚0.8355356209256031 "loss/total_loss‚8.953465068442 $ _timestamp‚1.784276315866087e+09 0loss/FRt2/assignment_loss‚1.1604319787025452 2loss/SRqqt2/assignment_loss‚0.5421868913383916 1loss/SRqqt1/detection_loss‚0.9715563005321667 0loss/FBt1/detection_loss‚0.48771333010585677 epoch‚10 /loss/FRt2/detection_loss‚1.3140916009469046 _runtime‚3507.991431508 _step‚337 1loss/FBt1/assignment_loss‚0.03716814898921887 1loss/SRqqt2/detection_loss‚0.6961638013196825 trainer/global_step‚4099Ñ¿QÃ0“‡‚(:ˆ ÝÊçÒôȤ gpu.0.gpu‚92.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚227.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.4136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×sïo ˆ‚(:• ÝÊçÒ”Ùõ¦ proc.cpu.threads‚17 disk.dm-0.in‚ 195.22265625% disk.nvme0n1.out‚13186.6650390625 disk.drbd2165.in‚0 disk.rbd0.in‚0 cpu‚26.05541554074397 disk.sda3.out‚ 105.2109375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2737.10546875 disk.sda2.in‚0 disk.rbd1.out‚92.45068359375$ memory_percent‚95.46618461608887* proc.memory.percent‚16.705965995788574 network.sent‚ 6.42901e+06 disk.nvme0n1.in‚ 195.22265625 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 195.22265625 network.recv‚942118' disk./.usagePercent‚36.648935465375$ disk./.usageGB‚546.0956230163574 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0" disk.dm-0.out‚13186.6650390625 disk.rbd2.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 105.2109375 disk.dm-1.out‚ 105.2109375' disk.nvme0c0n1.out‚13186.6650390625 disk.rbd1.in‚ 0.01171875 disk.rbd3.out‚0' proc.memory.availableMB‚ 742.8203125þô£‘‰‚(:† ìÊçÒ›Æ÷£ gpu.0.gpu‚81.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äƒ1õŠ‚(:’ ìÊçÒš³¬¦* disk./.usagePercent‚36.648940073510616 disk.dm-1.out‚ 105.34765625 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 742.62890625 disk.dm-1.in‚0* proc.memory.percent‚16.705965995788574 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 105.34765625 disk.sda1.in‚0 disk.sda.out‚ 105.34765625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚195.9375 disk.sda1.out‚0$ memory_percent‚95.46735286712646 disk.rbd1.out‚92.52880859375" disk.dm-0.out‚13194.9873046875% disk.nvme0n1.out‚13194.9873046875 disk.drbd2165.out‚0$ disk./.usageGB‚546.0956916809082 disk.rbd1.in‚ 0.01171875 disk.rbd2.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚13194.9873046875 network.sent‚ 6.480816e+06 disk.sda3.in‚0 disk.dm-0.in‚195.9375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚195.9375# proc.memory.rssMB‚ 2737.10546875 cpu‚26.054001202905845 network.recv‚945864´ÀX’‹‚(:‡ ûÊçÒ±˜¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì(ÊòŒ‚(:‘ ûÊçÒ땧 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0% disk.nvme0n1.out‚13204.9423828125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 105.6015625# proc.memory.rssMB‚ 2737.10546875# disk./.usageGB‚546.095760345459 disk.sda.in‚0 disk.dm-1.out‚ 105.6015625" disk.dm-0.out‚13204.9423828125 network.sent‚ 6.48516e+06 network.recv‚949167 disk.sda.out‚ 105.6015625 disk.sda2.in‚0* proc.memory.percent‚16.705965995788574 cpu‚26.052438017082544 disk.nvme0n1.in‚ 196.765625' disk.nvme0c0n1.out‚13204.9423828125 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚92.52880859375 disk.rbd3.out‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 742.6171875 disk.nvme0c0n1.in‚ 196.765625 disk.dm-1.in‚0 disk.dm-0.in‚ 196.765625# memory_percent‚95.4674243927002* disk./.usagePercent‚36.648944681646235 disk.rbd2.out‚0 proc.cpu.threads‚17#öq2- _wandbruntime‚3547  _runtime‚3547Ç*H4ÉŽÃ ' lr-AdamW/pg1‚0.00043126758247557574 ' lr-AdamW/pg2‚0.00043126758247557574 trainer/global_step‚4149 % _timestamp‚1.7842763552406588e+09 _runtime‚3547.366487263 _step‚338Ä/mÃÎÈ % _timestamp‚1.7842763552406588e+09 _runtime‚3547.366487263 _step‚338 ' lr-AdamW/pg1‚0.00043126758247557574 ' lr-AdamW/pg2‚0.00043126758247557574 trainer/global_step‚4149ÒÜ/—2- _wandbruntime‚3547  _runtime‚3547ïarºà'‘ÚO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634767413121536 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630381780459520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4379296856171 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139012429154304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312855 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630381780459520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143446623393280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385991040512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7272831 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7272828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32883320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚631520501082219 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385649701376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16197231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4434175626752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302876 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385991040512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139012447766528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32883320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16196958 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32883322 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429533 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8924400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚631520143492969 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚634767771500032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419556 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634767771500032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630381763420160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8924130 _step‚339 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630381763420160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32883322 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419556 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143446518301184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚3547.371393094 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496éÀ˜’ßO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302876 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385991040512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139012447766528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32883320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16196958 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32883322 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429533 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8924400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚631520143492969 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚634767771500032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419556 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634767771500032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630381763420160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8924130 _step‚339 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630381763420160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32883322 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419556 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143446518301184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚3547.371393094 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634767413121536 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630381780459520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4379296856171 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139012429154304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312855 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630381780459520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143446623393280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385991040512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytesëßÑ.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7272831 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7272828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32883320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚631520501082219 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385649701376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16197231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4434175626752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438Ó¨XœÝ2“- _wandbruntime‚3547  _runtime‚3547ðCñ;é&”ãM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32890954 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630529021489152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380307910005 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143480972981248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚630529004449792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32890954 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386662328320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8926489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4435199443968 _runtime‚3547.980431881 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38428608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚630529004449792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630529021489152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚631667665218563 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329553 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38438599 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784276355854558e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38428608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚627287714897920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚340 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71329553 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71319560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32890952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143481075969024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634916025163776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4387003674624 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8926221 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16200794 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139045857912832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0?®aåî&•èM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630529021489152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380307910005 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143480972981248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚630529004449792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32890954 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386662328320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8926489 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4435199443968 _runtime‚3547.980431881 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38428608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚630529004449792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630529021489152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚631667665218563 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329553 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38438599 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784276355854558e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38428608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚627287714897920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚340 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71329553 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71319560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32890952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143481075969024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634916025163776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4387003674624 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8926221 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16200794 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139045857912832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71319560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32890954 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872Ô4Á72–- _wandbruntime‚3547  _runtime‚3547Ô³V0—€ epoch‚10 /loss/FRt1/assignment_loss‚1.845740071161654 2loss/SRqqt1/assignment_loss‚0.5896813530194457 /loss/FBt1/detection_loss‚0.7063012150573684 0loss/FRt2/assignment_loss‚1.0347548865679346 2loss/SRqqt2/assignment_loss‚0.5442443191473278 1loss/FBt1/assignment_loss‚0.05162848107377016 .loss/FRt1/detection_loss‚0.526558673065982 /loss/FRt2/detection_loss‚1.3676530016670991 1loss/SRqqt1/detection_loss‚1.0263391468892162 1loss/SRqqt2/detection_loss‚0.6749253514362561 %loss/total_loss‚9.311039813712217 _runtime‚3547.982509092 1loss/FBt2/assignment_loss‚0.05127375852150536 /loss/FBt2/detection_loss‚0.8919395561046575 trainer/global_step‚4149 % _timestamp‚1.7842763558571784e+09 _step‚341òSj# ˜… /loss/FRt1/assignment_loss‚1.845740071161654 2loss/SRqqt1/assignment_loss‚0.5896813530194457 /loss/FBt1/detection_loss‚0.7063012150573684 0loss/FRt2/assignment_loss‚1.0347548865679346 2loss/SRqqt2/assignment_loss‚0.5442443191473278 1loss/FBt1/assignment_loss‚0.05162848107377016 .loss/FRt1/detection_loss‚0.526558673065982 /loss/FRt2/detection_loss‚1.3676530016670991 1loss/SRqqt1/detection_loss‚1.0263391468892162 1loss/SRqqt2/detection_loss‚0.6749253514362561 %loss/total_loss‚9.311039813712217 _runtime‚3547.982509092 1loss/FBt2/assignment_loss‚0.05127375852150536 /loss/FBt2/detection_loss‚0.8919395561046575 trainer/global_step‚4149 % _timestamp‚1.7842763558571784e+09 _step‚341 epoch‚10ÕÍ]ƒ“™‚(:ˆ ŠËçÒ’áó¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰E~8)š‚(:ž ŠËçÒ¯À’ª disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 cpu‚26.051063419661862 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚92.59130859375( disk.nvme0c0n1.out‚13212.38134765625 disk.sda.in‚0 disk.nvme0n1.in‚ 197.48046875 disk.rbd0.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.648939817503084 disk.dm-0.in‚ 197.48046875# disk.dm-0.out‚13212.38134765625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.46921253204346& disk.nvme0n1.out‚13212.38134765625( proc.memory.availableMB‚ 742.32421875# proc.memory.rssMB‚ 2737.10546875$ disk./.usageGB‚546.0956878662109 disk.rbd1.in‚ 0.01171875 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.rbd0.out‚0 network.recv‚952119 disk.drbd2165.out‚0* proc.memory.percent‚16.705965995788574 network.sent‚ 6.488082e+06 disk.sda3.out‚ 105.8046875 disk.dm-1.out‚ 105.8046875" disk.nvme0c0n1.in‚ 197.48046875 disk.sda.out‚ 105.8046875Šjôý’›‚(:‡ ™ËçÒº¬¤ gpu.0.gpu‚41.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚254.805' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—o*œ‚(:” ™ËçÒðƒ¶¦ disk.rbd1.out‚92.59130859375# proc.memory.rssMB‚ 2737.10546875 cpu‚26.049663100702777 disk.rbd0.out‚0 network.sent‚ 6.556967e+06 disk.nvme0n1.in‚ 197.93359375 proc.cpu.threads‚17 network.recv‚964301 disk.dm-0.in‚ 197.93359375" disk.dm-0.out‚13217.8173828125* proc.memory.percent‚16.705965995788574 disk.rbd2.in‚0% disk.nvme0n1.out‚13217.8173828125 disk.sda1.out‚0 disk.rbd1.in‚ 0.01171875% proc.memory.availableMB‚ 742.34375* disk./.usagePercent‚36.648962602173626$ disk./.usageGB‚546.0960273742676' disk.nvme0c0n1.out‚13217.8173828125 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sda3.in‚0# memory_percent‚95.4690933227539 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 106.203125 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 106.203125 disk.dm-1.in‚0 disk.dm-1.out‚ 106.203125 disk.rbd3.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 197.93359375ñƒ™’‚(:‡ ¨ËçÒ¸Æò¤ gpu.0.gpu‚31.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b!Z %ž‚(:š ¨ËçÒˆ¦°§) disk./.usagePercent‚36.64895287388733$ disk./.usageGB‚546.0958824157715 disk.sda.out‚ 106.3359375 disk.sda2.out‚0 disk.sda3.out‚ 106.3359375 cpu‚26.04834194588209 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚92.73193359375 disk.drbd2165.out‚0 network.recv‚967459 disk.rbd2.out‚0" disk.nvme0c0n1.in‚ 198.30859375' proc.memory.availableMB‚ 742.3828125 disk.sda1.in‚0 disk.drbd2165.in‚0 network.sent‚ 6.559931e+06 disk.nvme0n1.in‚ 198.30859375# proc.memory.rssMB‚ 2737.10546875& disk.nvme0n1.out‚13224.37744140625( disk.nvme0c0n1.out‚13224.37744140625 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 198.30859375# disk.dm-0.out‚13224.37744140625 disk.rbd3.in‚0# memory_percent‚95.4688549041748* proc.memory.percent‚16.705965995788574 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 106.3359375 proc.cpu.threads‚17øÁ6·2Ÿ- _wandbruntime‚3587  _runtime‚3587܆^Å ¿ % _timestamp‚1.7842763950916219e+09 _runtime‚3587.218047711 _step‚342 % lr-AdamW/pg1‚0.000430190454109417 % lr-AdamW/pg2‚0.000430190454109417 trainer/global_step‚4199?oŸÊ¡Ä % _timestamp‚1.7842763950916219e+09 _runtime‚3587.218047711 _step‚342 % lr-AdamW/pg1‚0.000430190454109417 % lr-AdamW/pg2‚0.000430190454109417 trainer/global_step‚4199Ö¼WË»2¢- _wandbruntime‚3587  _runtime‚3587jþª¥N£ÚO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7360078 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚4199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436686542336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436345203200 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4429572793189 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38883985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637743814901760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634466720780288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33264922 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9028979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842763950924344e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38883985 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145169256616960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7360081 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_byûÄÁÄ’tes.all.freed‚145169151524864 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33264920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _runtime‚3587.218545441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642180160104960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637743831941120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38874008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634466737819648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16389311 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4485370924032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4485284444160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140683867080704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _runtime‚3587.218545441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642180160104960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637743831941120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38874008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634466737819648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16389311 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4485370924032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4485284444160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140683867080704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 3587.83817953 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38883060 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642328413761536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784276395712353e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4486395118080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4430924396913 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437699176448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437357830144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚637891072970752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚637891055931392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7361826 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639043815299071 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155612 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38883060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634613248491520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4430583846911 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚4199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33272552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437357830144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639044172888433 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16392819 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165605 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33272554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7361823 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38893051 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144’–)>L§çM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639044172888433 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16392819 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165605 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33272554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7361823 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38893051 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚642328772147200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _step‚344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚637891055931392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16393077 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 3587.83817953 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38883060 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642328413761536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784276395712353e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4486395118080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4430924396913 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437699176448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437357830144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_ííÓÚ¡pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚637891072970752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚637891055931392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7361826 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639043815299071 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155612 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38883060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634613248491520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4430583846911 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚4199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33272552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437357830144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ØÅbíß2¨- _wandbruntime‚3587  _runtime‚3587hû#« ©„ /loss/FBt2/detection_loss‚0.5715573864290491 trainer/global_step‚4199 % _timestamp‚1.7842763957145722e+09 0loss/FRt1/assignment_loss‚1.9507631686362592 0loss/FRt2/assignment_loss‚1.0272634820062287 1loss/SRqqt1/detection_loss‚1.1035736643199652 _step‚345 1loss/SRqqt2/assignment_loss‚0.549275373205992 /loss/FRt1/detection_loss‚0.5557819513218999 /loss/FRt2/detection_loss‚1.3348253699397956 %loss/total_loss‚8.980335155187252 epoch‚10 _runtime‚3587.839897353 2loss/SRqqt1/assignment_loss‚0.6772753088962933 2loss/FBt1/assignment_loss‚0.014950592927902084 1loss/SRqqt2/detection_loss‚0.6067300799719613 /loss/FBt1/detection_loss‚0.5876526806833237 3loss/FBt2/assignment_loss‚0.0006860968485822294kåʪ‰ _step‚345 1loss/SRqqt2/assignment_loss‚0.549275373205992 /loss/FRt1/detection_loss‚0.5557819513218999 /loss/FRt2/detection_loss‚1.3348253699397956 %loss/total_loss‚8.980335155187252 epoch‚10 _runtime‚3587.839897353 2loss/SRqqt1/assignment_loss‚0.6772753088962933 2loss/FBt1/assignment_loss‚0.014950592927902084 1loss/SRqqt2/detection_loss‚0.6067300799719613 /loss/FBt1/detection_loss‚0.5876526806833237 3loss/FBt2/assignment_loss‚0.0006860968485822294 /loss/FBt2/detection_loss‚0.5715573864290491 trainer/global_step‚4199 % _timestamp‚1.7842763957145722e+09 0loss/FRt1/assignment_loss‚1.9507631686362592 0loss/FRt2/assignment_loss‚1.0272634820062287 1loss/SRqqt1/detection_loss‚1.1035736643199652ÙPS¢«‚(:’ ·ËçÒÙ¼ï¥ gpu.0.gpu‚81.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.27' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.854000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&M6#¬‚(:˜ ·ËçÒ¬…¨ network.recv‚970862 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 495.83203125 proc.cpu.threads‚17 disk.rbd1.out‚92.73193359375# proc.memory.rssMB‚ 2737.10546875 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚26.046899140295075 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 106.640625* proc.memory.percent‚16.705965995788574$ disk./.usageGB‚546.0971527099609 disk.nvme0n1.in‚ 199.0234375 disk.sda1.in‚0 disk.dm-1.out‚ 106.640625 network.sent‚ 6.564981e+06* disk./.usagePercent‚36.649038124396235 disk.dm-0.in‚ 199.0234375 disk.rbd2.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 199.0234375( disk.nvme0c0n1.out‚13232.81298828125 disk.rbd1.in‚ 0.01171875# disk.dm-0.out‚13232.81298828125& disk.nvme0n1.out‚13232.81298828125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚96.97368144989014 disk.rbd3.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 106.640625dï!Í“­‚(:ˆ ÆËçÒ “ò¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚267.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚073ð®‚(:‹ ÆËçÒþ•Ò¥ network.sent‚ 6.628455e+06 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 107.0234375 disk.rbd1.in‚ 0.01171875* proc.memory.percent‚16.705965995788574$ disk./.usageGB‚546.0971641540527$ disk.nvme0c0n1.out‚ 13241.5234375 disk.sda2.out‚0 disk.rbd3.out‚0 network.recv‚975514" disk.nvme0n1.out‚ 13241.5234375 disk.sda.in‚0 disk.sda.out‚ 107.0234375 disk.dm-1.in‚0 disk.rbd1.out‚92.79443359375# proc.memory.rssMB‚ 2737.10546875) disk./.usagePercent‚36.64903889241884 disk.dm-0.in‚ 199.8515625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.97322845458984 proc.cpu.threads‚17 disk.nvme0n1.in‚ 199.8515625! disk.nvme0c0n1.in‚ 199.8515625 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.out‚ 13241.5234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 107.0234375% proc.memory.availableMB‚ 495.90625 cpu‚26.045512874152525 disk.rbd2.in‚0 disk.sda2.in‚0¦P`Å2¯- _wandbruntime‚3627  _runtime‚3627¾Ž©Ç°Á & lr-AdamW/pg1‚0.0004291008747123312 & lr-AdamW/pg2‚0.0004291008747123312 trainer/global_step‚4249 % _timestamp‚1.7842764351014302e+09 _runtime‚3627.228444284 _step‚346®¾IÌ±Æ % _timestamp‚1.7842764351014302e+09 _runtime‚3627.228444284 _step‚346 & lr-AdamW/pg1‚0.0004291008747123312 & lr-AdamW/pg2‚0.0004291008747123312 trainer/global_step‚4249ÚYx“#2²- _wandbruntime‚3627  _runtime‚3627Ýû(à'³ÚO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4536355388928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487040705024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7447331 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39338437 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72974980 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649592907088384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16580700 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646272443653985 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39338437 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142355323619328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚641792271413248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487382044160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9133108 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842764351021776e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33646520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649593265466880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33646522 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚347 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487040705024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16580436 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487382044160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142355305007104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4480189280097 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚645105883422720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚649593265466880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚645105866383360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33646522 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _runtime‚3627.228546117H¶çºå'´ßO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚645105866383360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33646522 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _runtime‚3627.228546117 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4536355388928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487040705024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7447331 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39338437 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72974980 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649592907088384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16580700 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646272443653985 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39338437 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142355323619328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚641792271413248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487382044160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9133108 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842764351021776e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33646520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649593265466880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33646522 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚347 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487040705024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16580436 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487382044160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142355305007104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4480189280097 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚645105883422720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚649593265466880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616Û¢¹ù92µ- _wandbruntime‚3627  _runtime‚3627™LÚQM ¶äM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39347503 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142388752377856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚641938782085120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9135137 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16584210 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39347503 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39337512 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646420322968941 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7449073 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4537374388224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4488053331968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488053331968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4481540883821 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73001657 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646419965379579 _step‚348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 _runtime‚3627.841081861 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001657 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645253124452352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146926108153856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4481200333819 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488394678272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649741519130624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842764357151816e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360Ù;lï&·éM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645253124452352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146926108153856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4481200333819 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488394678272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚649741519130624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842764357151816e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39347503 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142388752377856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚641938782085120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9135137 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16584210 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39347503 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39337512 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646420322968941 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7449073 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4537374388224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4488053331968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488053331968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4481540883821 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73001657 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646419965379579 _step‚348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 _runtime‚3627.841081861 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001657 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652467934904320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9237266 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34028120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4537736206848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652467917864960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144026761545728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4531146316895 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144026742933504 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652467917864960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821011 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657005654071808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538077545984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4530805767005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34028122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39782912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538077545984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4587443559ºð=936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34028122 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7534578 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148614297876992 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657006012450304˜ øä'ÆÞO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34028122 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7534578 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148614297876992 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657006012450304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚649117805006848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652467934904320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9237266 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34028120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4537736206848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚652467917864960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144026761545728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4531146316895 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144026742933504 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652467917864960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821011 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657005654071808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538077545984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4530805767005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34028122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39782912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538077545984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4587443559936ß‹9p÷2Ç- _wandbruntime‚3667  _runtime‚36677âf"ê&ÈäM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34035754 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652615158894592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39791964 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148648745805824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653796473049449 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837709 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7536326 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9239281 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7536323 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚657153907728384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚653796115460087 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649264298639360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34035754 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16775604 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4588555501568 trainer/global_step‚4299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4588456445952 _runtime‚3667.620882287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657154266114048 _step‚352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269iu1Kï&ÉéM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34035752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚657153907728384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚653796115460087 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚649264298639360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34035754 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16775604 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4588555501568 trainer/global_step‚4299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4588456445952 _runtime‚3667.620882287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657154266114048 _step‚352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34035754 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652615158894592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39791964 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148648745805824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653796473049449 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837709 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7536326 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9239281 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7536323 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Eúà'×ÚO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40247341 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664418401055232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656443338600448 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664418401055232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4588431708672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34409722 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647084 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4588773047808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4638534426624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40237364 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4588773047808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150336733898752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4638439558144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34409720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4581762803803 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150336620418048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145698199472128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40237364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9341296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40247341 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7621828 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚3706.995571563 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚659829986385920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚659829969346560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚664418759433728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16963127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34409720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4581422253913 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34409722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 % _timestamp‚1.7842765148695426e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4588431708672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0™{å'ØßO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40247341 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664418401055232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656443338600448 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664418401055232 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4588431708672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34409722 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647084 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4588773047808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4638534426624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40237364 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4588773047808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150336733898752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4638439558144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34409720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4581762803803 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150336620418048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145698199472128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40237364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9341296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40247341 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7621828 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚3706.995571563 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚659829986385920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚659829969346560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚664418759433728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16963127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34409720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4581422253913 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34409722 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 % _timestamp‚1.7842765148695426e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4588431708672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16966712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664567013097472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7623573 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842765154830704e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673761 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40246416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9343139 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74663768 WID™ØX eviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚659977210376192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661172623129957 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656589832232960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74673761 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7623576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4639462981120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚659977227415552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4639553648128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40256407 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16966976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34417352ÅÄô ï&ÛéM trainer/global_step‚4349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34417354 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4589785681920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4589785681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150371072599552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40256407 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚659977210376192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40246416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4589444335616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4582433307635 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9343400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚656589849272320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150371181878784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145731628230656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16966712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664567013097472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7623573 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842765154830704e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673761 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40246416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9343139 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74663768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚659977210376192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661172623129957 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656589832232960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74673761 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7623576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4639462981120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚659977227415552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4639553648128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40256407 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16966976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34417352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ä+S<22Ü- _wandbruntime‚3707  _runtime‚3707bnä˜Ý€ 0loss/FRt1/assignment_loss‚1.8063959354962678 2loss/SRqqt2/assignment_loss‚0.6004936658273778 $loss/total_loss‚9.44875086999934 2loss/FBt2/assignment_loss‚0.031968031319845776 trainer/global_step‚4349 _runtime‚3707.610062593 epoch‚10 0loss/FRt2/assignment_loss‚1.0421885220910148 1loss/SRqqt1/assignment_loss‚0.637459255793574 1loss/FBt1/assignment_loss‚0.01032738606700821 1loss/SRqqt2/detection_loss‚0.6771072599030844 /loss/FBt1/detection_loss‚0.6095671657268412 /loss/FRt1/detection_loss‚0.5778672538769765 /loss/FRt2/detection_loss‚1.2580143311502756 0loss/SRqqt1/detection_loss‚1.141731328918912 /loss/FBt2/detection_loss‚1.0556307338281639 % _timestamp‚1.7842765154846687e+09 _step‚357TèµÂ Þ… 2loss/SRqqt2/assignment_loss‚0.6004936658273778 $loss/total_loss‚9.44875086999934 2loss/FBt2/assignment_loss‚0.031968031319845776 trainer/global_step‚4349 _runtime‚3707.610062593 epoch‚10 0loss/FRt2/assignment_loss‚1.0421885220910148 1loss/SRqqt1/assignment_loss‚0.637459255793574 1loss/FBt1/assignment_loss‚0.01032738606700821 1loss/SRqqt2/detection_loss‚0.6771072599030844 /loss/FBt1/detection_loss‚0.6095671657268412 /loss/FRt1/detection_loss‚0.5778672538769765 /loss/FRt2/detection_loss‚1.2580143311502756 0loss/SRqqt1/detection_loss‚1.141731328918912 /loss/FBt2/detection_loss‚1.0556307338281639 % _timestamp‚1.7842765154846687e+09 _step‚357 0loss/FRt1/assignment_loss‚1.8063959354962678åmÍ™¡‘ß‚(:† ¯ÌçÒÑêë¤ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä1a]*à‚(:Ÿ ¯ÌçÒ¸Š“§ disk.sda2.out‚0$ disk./.usageGB‚548.7266731262207 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 203.65234375 network.sent‚ 6.796189e+06* disk./.usagePercent‚36.825507445846014 disk.dm-0.in‚ 203.65234375 disk.sda.out‚ 115.23828125 disk.dm-1.in‚0 disk.dm-1.out‚ 115.23828125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚0$ disk.nvme0n1.out‚15059.259765625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 203.65234375 disk.sda3.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2737.10546875! disk.dm-0.out‚15059.259765625 disk.rbd0.out‚0 disk.sda3.out‚ 115.23828125$ memory_percent‚96.98147773742676 cpu‚26.03638783617078 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚92.98193359375' proc.memory.availableMB‚ 494.5546875* proc.memory.percent‚16.705965995788574 network.recv‚ 1.008213e+06 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚15059.259765625æM+“á‚(:ˆ ¾ÌçÒ«û¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã\W6â‚(: ¾ÌçÒ´ŒŽ§ disk.rbd3.out‚0 network.recv‚ 1.012555e+06 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda.out‚115.625 disk.rbd3.in‚0) disk./.usagePercent‚36.82551871017752% disk.nvme0n1.out‚16019.5068359375 disk.dm-1.in‚0 cpu‚26.035119734718027 disk.rbd0.in‚0' disk.nvme0c0n1.out‚16019.5068359375 disk.rbd1.in‚ 0.01171875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 204.48046875 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 204.48046875 disk.dm-0.in‚ 204.48046875* proc.memory.percent‚16.705965995788574$ disk./.usageGB‚548.7268409729004 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚93.04443359375" disk.dm-0.out‚16019.5068359375# proc.memory.rssMB‚ 2737.10546875 network.sent‚ 6.84745e+06 disk.dm-1.out‚115.625# memory_percent‚96.9827651977539% proc.memory.availableMB‚ 494.34375 proc.cpu.threads‚17 disk.rbd2.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚115.625gWõŸ2ã- _wandbruntime‚3746  _runtime‚3746‚úwÆäÀ & lr-AdamW/pg1‚0.0004257581341235101 & lr-AdamW/pg2‚0.0004257581341235101 trainer/global_step‚4399 % _timestamp‚1.7842765548561342e+09 _runtime‚ 3746.98172862 _step‚358jß+qËåÅ trainer/global_step‚4399 % _timestamp‚1.7842765548561342e+09 _runtime‚ 3746.98172862 _step‚358 & lr-AdamW/pg1‚0.0004257581341235101 & lr-AdamW/pg2‚0.0004257581341235101æûÎ^@2æ- _wandbruntime‚3746  _runtime‚3746»ç­á'çÛO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34791322 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152059060432896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚667192020828160 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4639468549632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚667192037867520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚671831506417152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4632379290711 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152059184399360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚667192020828160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚667192037867520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40701793 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7709081 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123966464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4632038740821 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671831148038656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668400893895509 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639127210496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34791322 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚668401251484759 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493115 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101ßIâ èàO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75483136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40701793 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4639468549632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7709078 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚667192037867520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40701793 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7709081 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123966464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4632038740821 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671831148038656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668400893895509 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639127210496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34791322 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚668401251484759 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493115 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34791322 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152059060432896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚667192020828160 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4639468549632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚667192037867520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚671831506417152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4632379290711 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152059184399360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚667192020828160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚671979401695232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4640481183744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34798952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4640481183744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152093622745600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668548773210465 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17158092 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4640139837440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚667339261857792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509813 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4690556588544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499820 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34798954 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9447524 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7710823 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671979760080896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4690459630080 trainer/global_step‚4399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4640139837440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34798952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34798954 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147403066157056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633049794543 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 _step‚360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚667339278897152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9447269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147403047544832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152093507174912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668548415621103 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 æØï&ëéM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667339278897152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚671979401695232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4640481183744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34798952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4640481183744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152093622745600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668548773210465 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17158092 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4640139837440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚667339261857792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509813 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4690556588544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499820 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34798954 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9447524 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7710823 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚671979760080896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4690459630080 trainer/global_step‚4399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4640139837440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34798952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34798954 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147403066157056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633049794543 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 _step‚360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚667339278897152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9447269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147403047544832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152093507174912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668548415621103 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _runtime‚3747.595195734 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26533010601997375 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.15902665257453918 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7072197794914246 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34285661578178406 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959456324577332 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4883100986480713 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945805668830872 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6564326286315918 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6145642995834351 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2191159576177597 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6888099908828735 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20272329449653625 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05770105496048927 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 'particle/f_score‚0.6353987455368042 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9844327569007874 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7019082307815552 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6281803846359253 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 *jet/accuracy_1_of_3‚0.9249660968780518 particle/accuracy_2_of_5‚1 )loss/val_total_loss‚8.624499633344417 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9942439198493958 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05008125305175781 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.030769333243370056 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9837161302566528 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22469013929367065 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7438616156578064 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8596581220626831 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9108806848526001 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8943719267845154 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48190179467201233 6validation_average_jet_accuracy‚0.3866559863090515 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7528678774833679 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7436782121658325 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03837057948112488 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3142276406288147 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8020346760749817 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7503417730331421 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5067487955093384 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23259088397026062 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6564326286315918 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8104686737060547 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7044855952262878 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30821964144706726 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_2‚0.7997083067893982 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7817460894584656 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 jet/accuracy_4_of_5‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7501740455627441 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6471192240715027 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12472432106733322 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3908425271511078 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9921032190322876 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7337528467178345 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02482912689447403 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.694288432598114 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6645551323890686 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6931716203689575 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_3_of_3‚0.9994933605194092 particle/accuracy_3_of_4‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6185790300369263 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26326239109039307 epoch‚10 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6599100232124329 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955938458442688 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19922998547554016 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13728731870651245 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4923059046268463 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7521406412124634 +jet/accuracy_3_of_3‚0.06940988451242447 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5565354228019714 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.992536723613739 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030130505561828613 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03236549347639084 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666793823242 % _timestamp‚1.7842765725303707e+09Fù+Ãë=óå{ 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9837161302566528 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22469013929367065 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7438616156578064 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8596581220626831 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9108806848526001 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8943719267845154 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48190179467201233 6validation_average_jet_accuracy‚0.3866559863090515 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7528678774833679 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7436782121658325 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03837057948112488 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3142276406288147 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8020346760749817 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7503417730331421 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5067487955093384 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23259088397026062 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6564326286315918 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8104686737060547 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7044855952262878 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30821964144706726 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_2‚0.7997083067893982 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7817460894584656 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 jet/accuracy_4_of_5‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7501740455627441 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6471192240715027 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12472432106733322 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3908425271511078 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9921032190322876 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7337528467178345 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02482912689447403 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.694288432598114 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6645551323890686 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6931716203689575 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_3_of_3‚0.9994933605194092 particle/accuracy_3_of_4‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6185790300369263 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26326239109039307 epoch‚10 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6599100232124329 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955938458442688 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19922998547554016 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13728731870651245 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4923059046268463 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7521406412124634 +jet/accuracy_3_of_3‚0.06940988451242447 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5565354228019714 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.992536723613739 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030130505561828613 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03236549347639084 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666793823242 % _timestamp‚1.7842765725303707e+09 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3881077170372009 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6405465006828308 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6714097261428833 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8981481194496155 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2734636664390564 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26326239109039307 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21504433453083038 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6930374503135681 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996832549571991 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7378495931625366 +particle/sensitivity‚0.6285807490348816 +particle/specificity‚0.9043081402778625 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977155327796936 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850418567657471 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8008314967155457 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8004283905029297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2623274326324463 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2739792466163635 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23336932063102722 *jet/accuracy_2_of_4‚0.7898333668708801 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8666957020759583 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.328125 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7460947632789612 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 trainer/global_step‚4399 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912294745445251 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948947429656982 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31975868344306946 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.577072024345398 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958730697632 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029742198064923286 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6612992286682129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.031350426375865936 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 _runtime‚3764.656617525 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.154227614402771 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48656776547431946 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5613354444503784 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6834772825241089 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8596581220626831 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.037109747529029846 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02482912689447403 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 jet/accuracy_2_of_5‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14107176661491394 *jet/accuracy_1_of_4‚0.9782986640930176 ,jet/accuracy_4_of_4‚0.013899499550461769 (particle/accuracy‚0.8450948596000671 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9852041602134705 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3935185372829437 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6545559763908386 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_1_of_1‚0.3467603325843811 *jet/accuracy_2_of_3‚0.5795726180076599 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9898291230201721 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.799546480178833 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20872046053409576 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06974107027053833 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9796581268310547 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9904585480690002 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22118456661701202 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.14814814925193787 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +jet/accuracy_3_of_4‚0.08084343373775482 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928755164146423 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996832549571991 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7462118268013 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9833332896232605 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.459488183259964 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7807906866073608 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 jet/accuracy_5_of_5‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9220544695854187 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22055304050445557 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6227291226387024 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 2 Purity/*FRt*SRqqt*FBt/FRt_purity‚ 0.23095703125 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.851593017578125 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6505106687545776 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21820996701717377 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7602120637893677 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7753231525421143 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9356968998908997 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7506486177444458 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2497859001159668 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2544582188129425 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.993377149105072 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27326369285583496 jet/accuracy_3_of_5‚1 _step‚362 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26533010601997375 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.15902665257453918 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7072197794914246 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34285661578178406 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959456324577332 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4883100986480713 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945805668830872 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6564326286315918 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6145642995834351 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2191159576177597 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6888099908828735 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20272329449653625 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05770105496048927 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 'particle/f_score‚0.6353987455368042 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9844327569007874 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7019082307815552 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6281803846359253 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 *jet/accuracy_1_of_3‚0.9249660968780518 particle/accuracy_2_of_5‚1 )loss/val_total_loss‚8.624499633344417 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9942439198493958 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05008125305175781 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.030769333243370056 particle/accuracy_3_of_5‚1êÍÞ€ ‘ô‚(:† ÜÌçÒæÚº¥ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ’ø(õ‚(: ÜÌçÒï‹Ý§! disk.nvme0c0n1.in‚ 206.0234375 disk.sda2.out‚0$ memory_percent‚97.46847152709961 network.sent‚ 6.926147e+06 disk.rbd2.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875 cpu‚26.05005534734492 disk.nvme0n1.in‚ 206.0234375 disk.sda1.out‚0 disk.sda3.out‚ 116.41796875 disk.dm-0.in‚ 206.0234375" disk.dm-0.out‚16037.5009765625 disk.rbd0.out‚0 disk.sda.out‚ 116.41796875 disk.dm-1.in‚0# proc.memory.rssMB‚ 2816.93359375 network.recv‚ 1.020501e+06) disk./.usagePercent‚36.82556018339807 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚16037.5009765625 disk.sda3.in‚0 disk.rbd1.out‚93.16943359375$ disk./.usageGB‚548.7274589538574 disk.rbd3.in‚0 disk.dm-1.out‚ 116.41796875 disk.rbd3.out‚0& proc.memory.availableMB‚ 414.765625 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.193198204040527 proc.cpu.threads‚17% disk.nvme0n1.out‚16037.5009765625ê‘Êåžö‚(:“ ëÌçҌ榤 gpu.0.gpu‚54.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚193.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.613400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05*„¥)÷‚(:ž ëÌçÒ̖צ disk.rbd0.out‚0 disk.dm-0.in‚ 206.73828125& disk.nvme0n1.out‚16292.00244140625) disk./.usagePercent‚36.82540069070423 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0$ memory_percent‚95.45783996582031$ proc.memory.availableMB‚744.1875 cpu‚26.04771752806561 disk.sda1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚16292.00244140625# disk.dm-0.out‚16292.00244140625 proc.cpu.threads‚17 network.sent‚ 6.97311e+06 network.recv‚ 1.02521e+06 disk.sda1.out‚0 disk.dm-1.out‚ 116.66015625 disk.rbd1.in‚ 0.01171875 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 206.73828125" proc.memory.rssMB‚ 2760.8515625" disk.nvme0c0n1.in‚ 206.73828125 disk.sda.out‚ 116.66015625 disk.rbd1.out‚93.16943359375* proc.memory.percent‚16.850900650024414 disk.rbd2.in‚0 disk.rbd2.out‚0$ disk./.usageGB‚548.7250823974609 disk.rbd0.in‚0 disk.sda3.out‚ 116.66015625Ãð“ø‚(:ˆ úÌçÒÌô¥£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÓ|0ù‚(:¥ úÌçÒò„ö¤* disk./.usagePercent‚36.649144623530475 disk.rbd3.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 206.73828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 116.80859375 disk.rbd0.in‚0 disk.sda.out‚ 116.80859375 network.sent‚ 6.976074e+06 network.recv‚ 1.028798e+06 disk.dm-0.in‚ 206.73828125 disk.nvme0n1.in‚ 206.73828125 disk.dm-1.out‚ 116.80859375 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚16541.56396484375 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2760.8515625 disk.rbd1.in‚ 0.01171875# disk.dm-0.out‚16541.56396484375( disk.nvme0c0n1.out‚16541.56396484375 disk.sda3.in‚0$ memory_percent‚95.45862674713135* proc.memory.percent‚16.850900650024414 cpu‚26.04644943577322 proc.cpu.threads‚17$ disk./.usageGB‚546.0987396240234 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 744.05859375 disk.rbd1.out‚93.23193359375 disk.sda.in‚0¥K¾2ú- _wandbruntime‚3805  _runtime‚3805M‰þÉûà % _timestamp‚1.7842766137944272e+09 _runtime‚3805.920014693 _step‚363 ' lr-AdamW/pg1‚0.00042461945834853875 ' lr-AdamW/pg2‚0.00042461945834853875 trainer/global_step‚4449çvÜÎüÈ ' lr-AdamW/pg1‚0.00042461945834853875 ' lr-AdamW/pg2‚0.00042461945834853875 trainer/global_step‚4449 % _timestamp‚1.7842766137944272e+09 _runtime‚3805.920014693 _step‚363ë^Šl¤2ý- _wandbruntime‚3805  _runtime‚3805aB+3þÚO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35293588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚676483638960128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149056968812544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚673003069001728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚681191553259008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17377540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677703784504581 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681191194880512 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681191553259008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4758866778112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153815854202880 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚364 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4707555920384 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681191194880512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153815751207936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4700715502853 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35293586 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4707897259520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4707555920384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4449 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673003051962368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3805.921401662 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7842766137954829e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚676483655999488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9578692 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4758782395392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35293588 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976Þ}ùEå'ÿßO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚676483638960128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149056968812544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚673003069001728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚681191553259008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17377540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677703784504581 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681191194880512 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681191553259008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4758866778112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153815854202880 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚364 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4707555920384 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681191194880512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153815751207936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4700715502853 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35293586 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4707897259520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4707555920384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4449 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673003051962368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3805.921401662 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7842766137954829e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚676483655999488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9578692 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4758782395392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35293588 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35293588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ì—ì¬2€- _wandbruntime‚3806  _runtime‚3806{l¦~é&ãM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4708909893632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4701726556687 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚676630897029120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17381640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153850191453696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚4449 _runtime‚3806.533972306 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9581044 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673149579673600 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4701386006685 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚676630897029120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚681339806922752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35301218 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153850302830080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41225647 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚676630879989760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149090416183296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677851306230287 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4759793882624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41225647 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17381378 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚677850948640925 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681339448537088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7800593 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4759886646784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35301220 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9580785 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35301218 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7800596 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673149562634240 $ _timestamp‚1.784276614408068e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41235638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4708568547328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438±L£°0 ‚èM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41235638 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4708568547328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4708909893632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4701726556687 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚676630897029120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17381640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153850191453696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚4449 _runtime‚3806.533972306 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9581044 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673149579673600 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4701386006685 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚676630897029120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚681339806922752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35301218 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153850302830080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41225647 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚676630879989760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149090416183296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677851306230287 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4759793882624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41225647 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17381378 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚677850948640925 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681339448537088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7800593 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4759886646784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35301220 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9580785 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35301218 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7800596 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673149562634240 $ _timestamp‚1.784276614408068e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0íõàuæ2ƒ- _wandbruntime‚3806  _runtime‚3806¡Ê(q„ü /loss/FRt1/detection_loss‚0.5457826952846676 1loss/SRqqt1/detection_loss‚0.9898049358466486 /loss/FRt1/assignment_loss‚1.896420405502405 1loss/SRqqt1/assignment_loss‚0.593103989221555 2loss/FBt2/assignment_loss‚0.037109468731981766 /loss/FBt2/detection_loss‚0.9242902226172961 2loss/FBt1/assignment_loss‚0.009875229667957258 .loss/FRt2/detection_loss‚1.242303486405249 0loss/FBt1/detection_loss‚0.46020654799448774 $ _timestamp‚1.784276614411196e+09 _runtime‚ 3806.53654293 /loss/FRt2/assignment_loss‚1.068305764149229 /loss/SRqqt2/detection_loss‚0.67477806285968 %loss/total_loss‚9.017465502110152 epoch‚11 trainer/global_step‚4449 _step‚366 1loss/SRqqt2/assignment_loss‚0.575484693828995äÞJƒ… /loss/FRt1/assignment_loss‚1.896420405502405 1loss/SRqqt1/assignment_loss‚0.593103989221555 2loss/FBt2/assignment_loss‚0.037109468731981766 /loss/FBt2/detection_loss‚0.9242902226172961 2loss/FBt1/assignment_loss‚0.009875229667957258 .loss/FRt2/detection_loss‚1.242303486405249 0loss/FBt1/detection_loss‚0.46020654799448774 $ _timestamp‚1.784276614411196e+09 _runtime‚ 3806.53654293 /loss/FRt2/assignment_loss‚1.068305764149229 /loss/SRqqt2/detection_loss‚0.67477806285968 %loss/total_loss‚9.017465502110152 epoch‚11 trainer/global_step‚4449 _step‚366 1loss/SRqqt2/assignment_loss‚0.575484693828995 /loss/FRt1/detection_loss‚0.5457826952846676 1loss/SRqqt1/detection_loss‚0.9898049358466486îXOh9“†‚(:ˆ ‰ÍçÒ«¹µ¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QHAH&‡‚(:› ‰ÍçÒóÆç§ network.sent‚ 6.981143e+06 disk.rbd3.out‚0 disk.dm-0.out‚16556.67578125 disk.sda2.out‚0" memory_percent‚95.450758934021 proc.cpu.threads‚17" proc.memory.rssMB‚ 2760.8515625* disk./.usagePercent‚36.649369142138006 disk.nvme0n1.in‚ 207.56640625 disk.sda2.in‚0 disk.rbd3.in‚0# disk.nvme0n1.out‚16556.67578125 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 118.48828125* proc.memory.percent‚16.850900650024414 disk.sda3.out‚ 118.48828125 disk.dm-0.in‚ 207.56640625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 207.56640625 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚16556.67578125( proc.memory.availableMB‚ 745.34765625 network.recv‚ 1.032333e+06$ disk./.usageGB‚546.1020851135254 disk.sda.in‚0 cpu‚26.045060775122465 disk.rbd1.out‚93.23193359375 disk.rbd0.in‚0 disk.sda.out‚ 118.48828125pZø“ˆ‚(:ˆ ˜ÍçÒ¤÷ߥ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëav)#‰‚(:˜ ˜ÍçÒ£³ï§ network.sent‚ 7.046517e+06 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 119.12890625 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 208.28125' disk.nvme0c0n1.out‚16564.8017578125 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚93.29443359375" disk.dm-0.out‚16564.8017578125 disk.nvme0c0n1.in‚ 208.28125( proc.memory.availableMB‚ 745.51171875 network.recv‚ 1.036804e+06 disk.sda3.in‚0 disk.sda3.out‚ 119.12890625% disk.nvme0n1.out‚16564.8017578125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.44975757598877 proc.cpu.threads‚17 disk.rbd3.out‚0$ disk./.usageGB‚546.1020278930664 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 119.12890625 disk.dm-0.in‚ 208.28125* proc.memory.percent‚16.850900650024414 cpu‚26.04385175348516) disk./.usagePercent‚36.64936530202499 disk.rbd2.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2760.85156253*ʉ’Š‚(:‡ §ÍçÒéºÀ¤ gpu.0.gpu‚17.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚207.404' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.4808 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç¿N+‹‚(:  §ÍçÒ‡¯ú¦ disk.sda2.out‚0 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.850900650024414 proc.cpu.threads‚17 disk.rbd2.out‚0& disk.nvme0n1.out‚16587.82177734375 disk.sda1.in‚0 disk.sda.out‚ 121.62890625( proc.memory.availableMB‚ 745.23828125 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 121.62890625 disk.sda3.out‚ 121.62890625 network.recv‚ 1.040501e+06$ disk./.usageGB‚546.1066474914551# disk.dm-0.out‚16587.82177734375 disk.nvme0n1.in‚ 209.109375 disk.sda1.out‚0 disk.rbd1.out‚93.29443359375$ memory_percent‚95.45142650604248* disk./.usagePercent‚36.649675327148955 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 209.109375" proc.memory.rssMB‚ 2760.8515625( disk.nvme0c0n1.out‚16587.82177734375 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 cpu‚26.042601750382296 network.sent‚ 7.049481e+06 disk.dm-0.in‚ 209.109375 disk.sda.in‚0 disk.sda3.in‚0ß‘Í2Œ- _wandbruntime‚3845  _runtime‚3845ƒó7ÇÁ & lr-AdamW/pg1‚0.0004234686893572879 & lr-AdamW/pg2‚0.0004234686893572879 trainer/global_step‚4499 % _timestamp‚1.7842766538400438e+09 _runtime‚3845.965647474 _step‚367^:ürÌŽÆ trainer/global_step‚4499 % _timestamp‚1.7842766538400438e+09 _runtime‚3845.965647474 _step‚367 & lr-AdamW/pg1‚0.0004234686893572879 & lr-AdamW/pg2‚0.0004234686893572879ï«æTW2- _wandbruntime‚3845  _runtime‚3845…ÔÉà'ÚO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155538387563008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683845707481088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35675186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _runtime‚3845.966963659 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚683845690441728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚688604300242432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4758251422208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41671047 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4751331989761 % _timestamp‚1.7842766538411515e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35675186 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚4499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683845690441728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17569404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17569135 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155538492655104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚688603941863936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41671047 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688603941863936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346233 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4758592761344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4758592761344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4809980824064 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35675188 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688603941863936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346233 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4758592761344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4758592761344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4809980824064 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35675188 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35675186 F@DeviceStatsMonitor.on_train_batl‹à»Qch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚4499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683845690441728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17569404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17569135 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155538492655104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚688603941863936ð$Ò‡2’- _wandbruntime‚3846  _runtime‚3846}Y»Lè&“âM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35682818 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155572840313856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4752343043595 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚3846.580425473 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7887843 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9685093 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4759605395456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150761835497472 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688752553906176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17572936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4759605395456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 # _timestamp‚1.78427665445449e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚688752553906176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35682818 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155572945398784 trainer/global_step‚4499 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41690090 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41680099 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4811004816384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685227098721433 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41680099 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41690090 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683992931471360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35682820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4759264049152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4752002493593 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚685227456310795 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683992931471360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35682820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9685348 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚680475096227840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683992948510720 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17573194 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7887846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150761854109696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4759264049152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚680475113267200 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚3846.580425473 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7887843 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9685093 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4759605395456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150761835497472 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688752553906176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17572936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4759605395456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 # _timestamp‚1.78427665445449e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚688752553906176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35682818 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155572945398784 trainer/global_step‚4499 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41690090 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41680099 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4811004816384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚685227098721433 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41680099 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41690090 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683992931471360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35682820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4759264049152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4752002493593 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚685227456310795 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683992931471360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35682820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9685348 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚680475096227840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683992948510720 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17573194 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7887846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150761854109696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4759264049152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚680475113267200 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ñßm2•- _wandbruntime‚3846  _runtime‚3846±1g –„ 2loss/SRqqt1/assignment_loss‚0.6324898339626727 4loss/FBt2/assignment_loss‚3.6060588941566607e-06 /loss/FRt1/detection_loss‚0.5640150804180525 0loss/SRqqt1/detection_loss‚0.957282961509599 /loss/FBt2/detection_loss‚0.9616233603252718 trainer/global_step‚4499 _runtime‚3846.582429044 0loss/FRt1/assignment_loss‚1.7846795006040443 2loss/SRqqt2/assignment_loss‚0.5926857381042494 /loss/FRt2/detection_loss‚1.3222671855922643 1loss/SRqqt2/detection_loss‚0.6379873354753559 epoch‚11 /loss/FRt2/assignment_loss‚1.050946473631335 2loss/FBt1/assignment_loss‚0.018063312406355194 %loss/total_loss‚9.055386014526583 _step‚370 /loss/FBt1/detection_loss‚0.5333416264384893 % _timestamp‚1.7842766544570594e+09²š—‰ _runtime‚3846.582429044 0loss/FRt1/assignment_loss‚1.7846795006040443 2loss/SRqqt2/assignment_loss‚0.5926857381042494 /loss/FRt2/detection_loss‚1.3222671855922643 1loss/SRqqt2/detection_loss‚0.6379873354753559 epoch‚11 /loss/FRt2/assignment_loss‚1.050946473631335 2loss/FBt1/assignment_loss‚0.018063312406355194 %loss/total_loss‚9.055386014526583 _step‚370 /loss/FBt1/detection_loss‚0.5333416264384893 % _timestamp‚1.7842766544570594e+09 2loss/SRqqt1/assignment_loss‚0.6324898339626727 4loss/FBt2/assignment_loss‚3.6060588941566607e-06 /loss/FRt1/detection_loss‚0.5640150804180525 0loss/SRqqt1/detection_loss‚0.957282961509599 /loss/FBt2/detection_loss‚0.9616233603252718 trainer/global_step‚4499ò Áõs›˜‚(: ¶ÍçÒ¥ˆ™¥ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚295.503' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.10059999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ßÖo*™‚(:Ÿ ¶ÍçÒŸË⦠disk.rbd3.out‚0# disk.dm-0.out‚16597.47412109375 disk.sda1.in‚0) disk./.usagePercent‚36.64958930861745 disk.sda1.out‚0 disk.dm-1.out‚ 124.36328125 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2760.8515625 disk.dm-0.in‚ 209.109375& disk.nvme0n1.out‚16597.47412109375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 209.109375 cpu‚26.041350937844815$ disk./.usageGB‚546.1053657531738 disk.rbd2.out‚0 disk.nvme0n1.in‚ 209.109375( disk.nvme0c0n1.out‚16597.47412109375 disk.sda.in‚0 disk.sda.out‚ 124.36328125* proc.memory.percent‚16.850900650024414 network.recv‚ 1.043631e+06 disk.drbd2165.in‚0 disk.rbd1.out‚93.35693359375 network.sent‚ 7.067575e+06 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚95.45156955718994 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 124.36328125 disk.rbd1.in‚ 0.01171875( proc.memory.availableMB‚ 745.21484375™Ïžš‚(:“ ÅÍçҡϤ gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-¦0›‚(:¥ ÅÍçÒáð‰§ disk.sda3.out‚ 126.23046875 disk.rbd3.in‚0" proc.memory.rssMB‚ 2760.8515625 cpu‚26.040102762193676 disk.rbd0.in‚0 network.recv‚ 1.048337e+06 disk.nvme0n1.in‚ 209.82421875 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 209.82421875) disk./.usagePercent‚36.64966918296814& disk.nvme0n1.out‚16618.92041015625" disk.nvme0c0n1.in‚ 209.82421875 disk.dm-1.in‚0 disk.rbd1.out‚93.35693359375$ memory_percent‚95.45118808746338( proc.memory.availableMB‚ 745.27734375( disk.nvme0c0n1.out‚16618.92041015625# disk.dm-0.out‚16618.92041015625 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 126.23046875 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.dm-1.out‚ 126.23046875* proc.memory.percent‚16.850900650024414 proc.cpu.threads‚17 network.sent‚ 7.117541e+06$ disk./.usageGB‚546.1065559387207 disk.rbd2.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0¨#“œ‚(:ˆ ÔÍçÒû–ï¤ gpu.0.gpu‚92.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/-•)‚(:ž ÔÍçÒø¯—§ disk.dm-1.out‚ 126.640625 disk.rbd1.out‚93.41943359375 disk.dm-0.in‚ 210.65234375 disk.rbd2.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.850900650024414# disk./.usageGB‚546.106575012207 disk.sda3.out‚ 126.640625 disk.rbd3.in‚0( disk.nvme0c0n1.out‚16625.89306640625& disk.nvme0n1.out‚16625.89306640625 disk.sda1.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 745.2734375 cpu‚26.038739769593917 network.sent‚ 7.122749e+06 disk.sda.out‚ 126.640625 disk.dm-1.in‚0# disk.dm-0.out‚16625.89306640625 disk.rbd2.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 1.051611e+06 disk.nvme0n1.in‚ 210.65234375" disk.nvme0c0n1.in‚ 210.65234375$ memory_percent‚95.45121192932129" proc.memory.rssMB‚ 2760.8515625 proc.cpu.threads‚17* disk./.usagePercent‚36.649670463005805 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0W Û2ž- _wandbruntime‚3885  _runtime‚3885…áùpÅŸ¿ & lr-AdamW/pg1‚0.0004223059010614994 & lr-AdamW/pg2‚0.0004223059010614994 trainer/global_step‚4549 # _timestamp‚1.78427669384276e+09 _runtime‚3885.968403511 _step‚371ëO‰–Ê Ä & lr-AdamW/pg2‚0.0004223059010614994 trainer/global_step‚4549 # _timestamp‚1.78427669384276e+09 _runtime‚3885.968403511 _step‚371 & lr-AdamW/pg1‚0.0004223059010614994ó'™12¡- _wandbruntime‚3885  _runtime‚3885ä"‘¢ÙO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42135476 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691207758962688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _runtime‚3885.970486583 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚696017047225856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152399844665344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4809288263168 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78192264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696016688847360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4861212428288 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9786378 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4801607926779 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842766938435566e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17759726 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7973348 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36056786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚696016688847360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42125499 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17759989 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157261075705856 Iïä'£ÞO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78192264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696016688847360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4861212428288 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9786378 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4801607926779 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842766938435566e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17759726 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7973348 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36056786 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚696016688847360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42125499 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17759989 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157261075705856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42135476 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691207758962688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _runtime‚3885.970486583 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚696017047225856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152399844665344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4809288263168 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169ôË:û2¤- _wandbruntime‚3886  _runtime‚3886ÜŠ`†ê&¥äM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4810300897280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚696165300889600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691354999992320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4810300897280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚373 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198969 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17763581 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17763851 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9788488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4862235319808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692603248801941 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4809959550976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7975093 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 % _timestamp‚1.7842766944550045e+09 _runtime‚3886.580807698 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42134551 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚691354982952960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692603606391303 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42144542 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚687800646860800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36064420 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4809959550976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691354982952960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157295418076672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198969 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36064418 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152433292036096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4802959530503 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9788755 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696165300889600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691354999992320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42134551 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0k²û²›¦éM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17763581 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17763851 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9788488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4862235319808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692603248801941 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4809959550976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7975093 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 % _timestamp‚1.7842766944550045e+09 _runtime‚3886.580807698 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42134551 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚691354982952960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚692603606391303 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42144542 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚687800646860800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36064420 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4809959550976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691354982952960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157295418076672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198969 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36064418 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152433292036096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4802959530503 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9788755 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num.üÅT _device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696165300889600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691354999992320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42134551 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4810300897280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚696165300889600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691354999992320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4810300897280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚373 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198969 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169õ¿k£±2§- _wandbruntime‚3886  _runtime‚3886ÖåH ¨… _step‚374 0loss/FRt1/assignment_loss‚1.8478466107080669 /loss/FRt2/assignment_loss‚1.023560286387545 2loss/SRqqt1/assignment_loss‚0.5584200412081659 2loss/SRqqt2/assignment_loss‚0.5262475951395629 1loss/SRqqt1/detection_loss‚1.0335826542593438 0loss/FBt1/detection_loss‚0.43267857159671796 %loss/total_loss‚8.652973760761029 trainer/global_step‚4549 % _timestamp‚1.7842766944575071e+09 _runtime‚3886.582897283 2loss/FBt1/assignment_loss‚0.015604079671313737 /loss/FBt2/detection_loss‚0.7627860618416559 3loss/FBt2/assignment_loss‚0.0011280039952287718 /loss/FRt1/detection_loss‚0.5285390401492758 /loss/FRt2/detection_loss‚1.2499839561976192 1loss/SRqqt2/detection_loss‚0.6725968596065321 epoch‚11Ý2±2©Š /loss/FRt1/detection_loss‚0.5285390401492758 /loss/FRt2/detection_loss‚1.2499839561976192 1loss/SRqqt2/detection_loss‚0.6725968596065321 epoch‚11 _step‚374 0loss/FRt1/assignment_loss‚1.8478466107080669 /loss/FRt2/assignment_loss‚1.023560286387545 2loss/SRqqt1/assignment_loss‚0.5584200412081659 2loss/SRqqt2/assignment_loss‚0.5262475951395629 1loss/SRqqt1/detection_loss‚1.0335826542593438 0loss/FBt1/detection_loss‚0.43267857159671796 %loss/total_loss‚8.652973760761029 trainer/global_step‚4549 % _timestamp‚1.7842766944575071e+09 _runtime‚3886.582897283 2loss/FBt1/assignment_loss‚0.015604079671313737 /loss/FBt2/detection_loss‚0.7627860618416559 3loss/FBt2/assignment_loss‚0.0011280039952287718öˆýÀדª‚(:ˆ ãÍçÒóÆõ¤ gpu.0.gpu‚93.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚204.451' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.8902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dÒˆ&«‚(:› ãÍçÒ•›“§$ memory_percent‚96.95448875427246 disk.rbd3.in‚0$ disk.nvme0n1.out‚16635.630859375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 211.3671875 disk.sda.out‚ 127.80859375) disk./.usagePercent‚36.64935634176129 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2760.8515625$ disk./.usageGB‚546.1018943786621 disk.sda1.out‚0 disk.sda3.out‚ 127.80859375 disk.dm-1.out‚ 127.80859375 cpu‚26.037452247710807 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚16635.630859375 disk.sda3.in‚0* proc.memory.percent‚16.850900650024414 proc.cpu.threads‚17 network.recv‚ 1.055192e+06' proc.memory.availableMB‚ 498.9765625! disk.dm-0.out‚16635.630859375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 211.3671875 disk.rbd2.out‚0 disk.nvme0n1.in‚ 211.3671875 disk.rbd1.in‚ 0.01171875 network.sent‚ 7.127826e+06 disk.rbd0.out‚0 disk.rbd1.out‚93.41943359375. Ç“¬‚(:ˆ òÍçÒÆ•š¥ gpu.0.gpu‚70.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Äq*&­‚(:› òÍçÒïꨧ$ disk./.usageGB‚546.0992393493652 disk.nvme0n1.in‚ 212.1953125& disk.nvme0n1.out‚16648.40576171875 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0% proc.memory.availableMB‚ 499.09375" proc.memory.rssMB‚ 2760.8515625 disk.drbd2165.in‚0 network.recv‚ 1.059475e+06 disk.dm-0.in‚ 212.1953125 disk.rbd0.in‚0* proc.memory.percent‚16.850900650024414 disk.rbd3.in‚0 cpu‚26.03636119580204 disk.sda3.out‚ 129.6953125 disk.rbd1.in‚ 0.01171875 disk.rbd2.in‚0 disk.sda.in‚0 network.sent‚ 7.192966e+06! disk.nvme0c0n1.in‚ 212.1953125( disk.nvme0c0n1.out‚16648.40576171875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.out‚0 disk.sda.out‚ 129.6953125 disk.rbd1.out‚93.54443359375# disk.dm-0.out‚16648.40576171875 disk.dm-1.in‚0$ memory_percent‚96.95377349853516 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 129.6953125 disk.sda1.in‚0) disk./.usagePercent‚36.64917816051746´Û„2®- _wandbruntime‚3925  _runtime‚3925Ü¡·#ɯà ' lr-AdamW/pg1‚0.00042113116814489253 ' lr-AdamW/pg2‚0.00042113116814489253 trainer/global_step‚4599 % _timestamp‚1.7842767338179579e+09 _runtime‚3925.943568041 _step‚3758’eÎ°È ' lr-AdamW/pg1‚0.00042113116814489253 ' lr-AdamW/pg2‚0.00042113116814489253 trainer/global_step‚4599 % _timestamp‚1.7842767338179579e+09 _runtime‚3925.943568041 _step‚375÷,«½§2±- _wandbruntime‚3925  _runtime‚3925iž ^ß'²ÙO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36438388 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4859983764992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚698569810444288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚703429435830784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36438386 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36438388 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9889774 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703429435830784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚698569810444288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚698569793404928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9890040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42589928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018337 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42589928 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42579951 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42579951 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8060598 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚699831877156855 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154071282591744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842767338190455e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4859983764992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚699832234746105 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚703429794209280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚698569793404928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4852564963577 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4859642425856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17950372 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694979669782528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚694979652743168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚703429794209280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912297745920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚4599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36438386 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17950641 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158983598949888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4912209168896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4859642425856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17950372 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694979669782528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚694979652743168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚703429794209280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4912297745920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚4599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36438386 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17950641 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158983598949888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4912209168896 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703429435830784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚698569810444288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚698569793404928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9890040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42589928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018337 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42589928 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42579951 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42579951 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8060598 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚699831877156855 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154071282591744 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842767338190455e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4859983764992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚699832234746105 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚703429794209280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚698569793404928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4852564963577 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17954190 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062346 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154104729962496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154104711350272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4853576017411 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3926.558050734 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36446018 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9891847 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚703578047873024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842¨s]521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚698717034434560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17954473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚699979756471811 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42598994 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚695126163415040 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4860996399104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36446020 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚695126180454400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4853235467409 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062343 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36446020 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚698717034434560 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42598994 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159017941992448 % _timestamp‚1.7842767344321563e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698717051473920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018049174528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36446018 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚698717051473920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860655052800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42589003 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913319212032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4913230642176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699979398882449 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚4599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9892127 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79035021 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169èa†Ÿï&¶éM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚698717034434560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17954473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚699979756471811 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42598994 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚695126163415040 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4860996399104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36446020 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚695126180454400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4853235467409 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062343 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36446020 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚698717034434560 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42598994 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159017941992448 % _timestamp‚1.7842767344321563e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698717051473920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018049174528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36446018 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚698717051473920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860655052800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42589003 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913319212032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4913230642176 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699979398882449 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚4599 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9892127 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79035021 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17954190 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062346 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154104729962496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154104711350272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4853576017411 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3926.558050734 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36446018 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9891847 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚703578047873024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014ù´Y/2·- _wandbruntime‚3926  _runtime‚3926̉P„¸€ 0loss/FRt2/assignment_loss‚1.0069542713463306 1loss/SRqqt1/assignment_loss‚0.577268730857126 1loss/SRqqt1/detection_loss‚1.0558451171235546 1loss/SRqqt2/detection_loss‚0.6532705880761818 /loss/FBt2/detection_loss‚0.7692980773281306 %loss/total_loss‚9.065370778208463 % _timestamp‚1.7842767344351525e+09 1loss/FBt1/assignment_loss‚0.02394675954734167 0loss/FBt2/assignment_loss‚0.0867108695752888 /loss/FRt1/detection_loss‚0.5567041905229874 _runtime‚3926.560504627 _step‚378 2loss/SRqqt2/assignment_loss‚0.6190565383801144 /loss/FRt2/detection_loss‚1.2400686925019337 /loss/FBt1/detection_loss‚0.6466129135145378 epoch‚11 trainer/global_step‚4599 0loss/FRt1/assignment_loss‚1.8296340294349354XÞç^ ¹… 2loss/SRqqt2/assignment_loss‚0.6190565383801144 /loss/FRt2/detection_loss‚1.2400686925019337 /loss/FBt1/detection_loss‚0.6466129135145378 epoch‚11 trainer/global_step‚4599 0loss/FRt1/assignment_loss‚1.8296340294349354 0loss/FRt2/assignment_loss‚1.0069542713463306 1loss/SRqqt1/assignment_loss‚0.577268730857126 1loss/SRqqt1/detection_loss‚1.0558451171235546 1loss/SRqqt2/detection_loss‚0.6532705880761818 /loss/FBt2/detection_loss‚0.7692980773281306 %loss/total_loss‚9.065370778208463 % _timestamp‚1.7842767344351525e+09 1loss/FBt1/assignment_loss‚0.02394675954734167 0loss/FBt2/assignment_loss‚0.0867108695752888 /loss/FRt1/detection_loss‚0.5567041905229874 _runtime‚3926.560504627 _step‚378ú°´F^’º‚(:‡ ÎçÒ³õ¬¦ gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚254.285' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.857 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FÂÀ3(»‚(: ÎçÒ“„Ψ disk.rbd3.in‚0 disk.dm-0.in‚ 212.1953125 disk.sda.out‚ 130.2265625 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚ 1.063031e+06) disk./.usagePercent‚36.64931896466129 disk.rbd3.out‚0$ disk./.usageGB‚546.1013374328613# disk.dm-0.out‚16657.96533203125 disk.drbd2165.out‚0$ memory_percent‚96.95367813110352& proc.memory.availableMB‚ 499.109375 disk.rbd2.out‚0& disk.nvme0n1.out‚16657.96533203125 disk.drbd2165.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚16657.96533203125 disk.dm-1.out‚ 130.2265625" proc.memory.rssMB‚ 2760.8515625* proc.memory.percent‚16.850900650024414 disk.nvme0n1.in‚ 212.1953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚93.54443359375 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 212.1953125 disk.sda.in‚0 disk.sda3.out‚ 130.2265625 cpu‚26.035200195158268 network.sent‚ 7.209072e+06 disk.rbd2.in‚0 disk.sda3.in‚0µ–Y“¼‚(:ˆ ÎçÒˆ£¤¥ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÊÉ (½‚(: ÎçÒÀÓʧ disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 212.91015625 network.sent‚ 7.261164e+06 disk.rbd2.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚546.0993537902832 disk.sda3.out‚ 130.578125 disk.rbd2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 130.578125 disk.sda3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚16665.90576171875 disk.sda.out‚ 130.578125& proc.memory.availableMB‚ 499.046875" proc.memory.rssMB‚ 2760.8515625 disk.rbd1.out‚93.60693359375 disk.dm-0.in‚ 212.91015625 disk.nvme0n1.in‚ 212.91015625& disk.nvme0n1.out‚16665.90576171875( disk.nvme0c0n1.out‚16665.90576171875* proc.memory.percent‚16.850900650024414 cpu‚26.03389165703081 disk.dm-1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚36.649185840743485 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0$ memory_percent‚96.95405960083008 network.recv‚ 1.067308e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0±Íþ‚(:’ ŸÎçÒ°À‡¤ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.30499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»±,¿‚(:¡ ŸÎçÒâ–«¦ network.recv‚ 1.07017e+06) disk./.usagePercent‚36.64949765792019 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 498.79296875* proc.memory.percent‚16.850900650024414& disk.nvme0n1.out‚16685.87255859375 disk.rbd3.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚16685.87255859375 disk.sda.out‚ 131.80078125 disk.sda3.in‚0 cpu‚26.03272070301436$ disk./.usageGB‚546.1040000915527 disk.rbd0.out‚0 disk.sda3.out‚ 131.80078125 disk.rbd1.out‚93.60693359375 disk.rbd2.out‚0 disk.dm-0.in‚ 213.73828125# disk.dm-0.out‚16685.87255859375 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 131.80078125 disk.rbd1.in‚ 0.01171875 network.sent‚ 7.2653e+06 disk.sda2.in‚0 disk.rbd3.out‚0$ memory_percent‚96.95560932159424" proc.memory.rssMB‚ 2760.8515625 proc.cpu.threads‚17 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 213.73828125 disk.sda2.out‚0 disk.nvme0n1.in‚ 213.73828125 disk.sda1.in‚0q\­#2À- _wandbruntime‚3965  _runtime‚3965_ÐW˜ÉÁà trainer/global_step‚4649 % _timestamp‚1.7842767738475096e+09 _runtime‚3965.973122168 _step‚379 ' lr-AdamW/pg1‚0.00041994456605836745 ' lr-AdamW/pg2‚0.00041994456605836745‰Í9ÎÂÈ trainer/global_step‚4649 % _timestamp‚1.7842767738475096e+09 _runtime‚3965.973122168 _step‚379 ' lr-AdamW/pg1‚0.00041994456605836745 ' lr-AdamW/pg2‚0.00041994456605836745û+4i2Ã- _wandbruntime‚3965  _runtime‚39650š,Cà'ÄÚO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚707208384826613 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚702305186336768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚380 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155742739130368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8147851 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8147848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4963380184576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚702305203376128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4910337927680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚707208027237363 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚4649 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3965.974500681 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4902840900595 % _timestamp‚1.7842767738487098e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚705931861925888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9993260 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160706012125696 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710842541192704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710842182814208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18141108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160706119314944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9993523 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4963291607552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18141374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710842182814208 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ÈoíáÅßO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚705931861925888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43034403 ?9DeviceStatsMonitor.on_train_batch_sàOFW'tart/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚705931844886528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710842182814208 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚707208384826613 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚702305186336768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚380 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155742739130368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8147851 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8147848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4963380184576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚702305203376128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4910337927680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚707208027237363 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚4649 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3965.974500681 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4902840900595 % _timestamp‚1.7842767738487098e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚705931861925888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9993260 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160706012125696 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710842541192704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710842182814208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18141108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160706119314944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9993523 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4963291607552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18141374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36827620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8149596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚702451697008640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710990436470784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9995612 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707355548962957 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚702451714048000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4903851954317 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36827618 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871073 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚4649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4911350554624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18145208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43043455 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚381 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4911350554624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706079085916160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4964308542464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710990794856448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18144935 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4911691900928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43043455 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160740567098368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4964399209472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871073 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36827620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4904192504319 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36827618 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43053446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚707355906552319 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9995342 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968aÊÉßï&ÈéM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871073 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚4649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4911350554624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18145208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43043455 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚381 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4911350554624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706079085916160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4964308542464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710990794856448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18144935 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4911691900928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43043455 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚160740567098368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4964399209472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871073 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36827620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4904192504319 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36827618 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43053446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚707355906552319 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9995342 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36827620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8149596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚702451697008640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710990436470784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9995612 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707355548962957 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚702451714048000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4903851954317 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36827618 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169ý ²;2É- _wandbruntime‚3966  _runtime‚3966¹¹’Êÿ /loss/FBt2/detection_loss‚0.7331784068872886 epoch‚11 _runtime‚3966.589484946 _step‚382 0loss/FRt1/assignment_loss‚1.8449984875640113 2loss/SRqqt1/assignment_loss‚0.4780723243609048 /loss/FRt2/detection_loss‚1.3525895660783231 .loss/FBt1/detection_loss‚0.533909139360356 1loss/FBt2/assignment_loss‚0.07425219549142119 #loss/total_loss‚8.7687576136083 trainer/global_step‚4649 % _timestamp‚1.7842767744641595e+09 0loss/FRt2/assignment_loss‚1.0026998439531647 2loss/SRqqt2/assignment_loss‚0.5556518241975215 /loss/FRt1/detection_loss‚0.5529616968566707 1loss/SRqqt2/detection_loss‚0.6812309296836786 1loss/FBt1/assignment_loss‚0.01734092729119155 1loss/SRqqt1/detection_loss‚0.9418722718837671}ï Ë„ 0loss/FRt2/assignment_loss‚1.0026998439531647 2loss/SRqqt2/assignment_loss‚0.5556518241975215 /loss/FRt1/detection_loss‚0.5529616968566707 1loss/SRqqt2/detection_loss‚0.6812309296836786 1loss/FBt1/assignment_loss‚0.01734092729119155 1loss/SRqqt1/detection_loss‚0.9418722718837671 /loss/FBt2/detection_loss‚0.7331784068872886 epoch‚11 _runtime‚3966.589484946 _step‚382 0loss/FRt1/assignment_loss‚1.8449984875640113 2loss/SRqqt1/assignment_loss‚0.4780723243609048 /loss/FRt2/detection_loss‚1.3525895660783231 .loss/FBt1/detection_loss‚0.533909139360356 1loss/FBt2/assignment_loss‚0.07425219549142119 #loss/total_loss‚8.7687576136083 trainer/global_step‚4649 % _timestamp‚1.7842767744641595e+09þ œÌ‚(:‘ ®ÎçÒÆ’ˆ¥ gpu.0.gpu‚26.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ãå:ÕÍ‚(:š ®ÎçÒºñˆ§ disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0% disk.nvme0n1.out‚17004.9169921875 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 214.453125$ memory_percent‚96.95577621459961" proc.memory.rssMB‚ 2760.8515625 cpu‚26.031531700336398 disk.dm-0.in‚ 214.453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.out‚0* disk./.usagePercent‚36.737601114783786 disk.sda3.out‚ 132.98046875 disk.dm-1.in‚0" disk.dm-0.out‚17004.9169921875& proc.memory.availableMB‚ 498.765625[3YÍP network.sent‚ 7.281375e+06 disk.dm-1.out‚ 132.98046875* proc.memory.percent‚16.850900650024414 disk.rbd2.out‚0# disk./.usageGB‚547.416805267334 disk.nvme0c0n1.in‚ 214.453125' disk.nvme0c0n1.out‚17004.9169921875 disk.sda.out‚ 132.98046875 disk.rbd1.out‚93.66943359375 network.recv‚ 1.073439e+06P+7“΂(:ˆ ½ÎçÒÌÎó¤ gpu.0.gpu‚35.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚274.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ý"Ï‚(:— ½ÎçÒ®ª—§ disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 215.28125 cpu‚26.03042176886993 proc.cpu.threads‚17 network.recv‚ 1.077853e+06 disk.rbd2.in‚0 disk.nvme0n1.in‚ 215.28125$ disk./.usageGB‚547.4169769287109 disk.sda1.in‚0' disk.nvme0c0n1.out‚18049.0380859375 disk.sda.in‚0 disk.sda.out‚ 133.52734375 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 215.28125 network.sent‚ 7.331351e+06 disk.rbd2.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 133.52734375 disk.rbd1.out‚93.66943359375" proc.memory.rssMB‚ 2760.8515625* proc.memory.percent‚16.850900650024414 disk.rbd0.in‚0 disk.dm-1.out‚ 133.52734375 disk.rbd1.in‚ 0.01171875" disk.dm-0.out‚18049.0380859375$ memory_percent‚96.95601463317871 disk.sda1.out‚0' proc.memory.availableMB‚ 498.7265625) disk./.usagePercent‚36.73761263512282% disk.nvme0n1.out‚18049.0380859375®ði ‘Ђ(:† ÌÎçÒÆÜå¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òXn('Ñ‚(:œ ÌÎçÒÓæú¦ disk.sda.in‚0 disk.rbd3.in‚0 cpu‚26.029139300777192 proc.cpu.threads‚17 disk.rbd3.out‚0 network.recv‚ 1.081146e+06& disk.nvme0n1.out‚19397.04443359375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 134.08203125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 134.08203125 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚93.73193359375 disk.rbd2.out‚0$ memory_percent‚96.95603847503662" proc.memory.rssMB‚ 2760.8515625$ disk./.usageGB‚547.4169998168945 disk.sda2.out‚0 disk.sda3.out‚ 134.08203125# disk.dm-0.out‚19397.04443359375) disk./.usagePercent‚36.73761417116803 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 215.28125 disk.sda2.in‚0 disk.dm-0.in‚ 215.28125 disk.nvme0n1.in‚ 215.28125 disk.sda1.out‚0 network.sent‚ 7.336557e+06 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚19397.04443359375 disk.rbd2.in‚0( proc.memory.availableMB‚ 498.72265625* proc.memory.percent‚16.850900650024414s£`”2Ò- _wandbruntime‚4005  _runtime‚4005rà3ÇÓÁ trainer/global_step‚4699 % _timestamp‚1.7842768136018326e+09 _runtime‚4005.727447756 _step‚383 & lr-AdamW/pg1‚0.0004187461710151589 & lr-AdamW/pg2‚0.0004187461710151589æÚ×.ÌÔÆ & lr-AdamW/pg2‚0.0004187461710151589 trainer/global_step‚4699 % _timestamp‚1.7842768136018326e+09 _runtime‚4005.727447756 _step‚383 & lr-AdamW/pg1‚0.0004187461710151589ÿˆb Ž2Õ- _wandbruntime‚4005  _runtime‚4005€LÂá'ÖÛO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚714584177317871 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43488855 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚718254929797632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚709630719930368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚713293913407488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709630736969728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162428724431360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚4699 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718254929797632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43498832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37201586 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4961033429504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚718255288176128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18332697 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5014456700416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37201588 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚713293896368128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43488855 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8235098 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10097599 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5014547374592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4005.732382072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚718255288176128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157414177056768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157414158444544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784276813602865e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18332955 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953457387503 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162428615144960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚713293913407488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10097854 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4961033429504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976‰ÏíSæ'×àO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚4699 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718254929797632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43498832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37201586 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4961033429504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚718255288176128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18332697 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5014456700416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37201588 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚713293896368128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43488855 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8235098 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10097599 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5014547374592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚4005.732382072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚718255288176128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157414177056768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157414158444544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784276813602865e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18332955 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953457387503 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162428615144960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚713293913407488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10097854 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4961033429504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚714584177317871 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43488855 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚718254929797632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚709630719930368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚713293913407488 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709630736969728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162428724431360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚384€zŠæÆ2Ø- _wandbruntime‚4006  _runtime‚4006 Rø ÙæM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4962387402752 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37209220 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43497907 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707125 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718403541839872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713441137397760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚713441154437120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5015466704896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4962046056448 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4699 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37209220 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954468441225 _runtime‚4006.341736558 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18336803 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18336532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚718403541839872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162463171575808>%dñ&ÚëM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37209220 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43497907 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707125 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718403541839872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713441137397760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚713441154437120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5015466704896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4962046056448 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4699 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37209220 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954468441225 _runtime‚4006.341736558 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18336803 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18336532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚718403541839872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162463171575808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4962387402752 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚716956253523968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725668035159552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5012070270464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159085596370944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721960327398379 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37583186 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164151188929024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716956270563328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4749 _runtime‚ 4045.69339004 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8322351 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5011728931328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5004414424301 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37583186 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5065573945856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43943307 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5011728931328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5004073874411 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725667676781056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720655947849728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43953284 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18523903 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5065483271680 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8322348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5012070270464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10201279 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526493 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10201552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720655947849728äìû (çàO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37583188 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720655964889088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43943307 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18523627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164151079642624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842768535638547e+09 _step‚388 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚716956253523968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725668035159552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5012070270464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159085596370944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721960327398379 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37583186 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164151188929024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716956270563328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4749 _runtime‚ 4045.69339004 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8322351 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5011728931328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5004414424301 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37583186 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5065573945856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43943307 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_trg$Ÿ¾ain_batch_start/active_bytes.small_pool.freed‚ 5011728931328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5004073874411 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725667676781056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720655947849728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43953284 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18523903 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5065483271680 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8322348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5012070270464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10201279 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526493 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10201552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720655947849728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43953284„)ÂÂ2è- _wandbruntime‚4046  _runtime‚4046¡ßQ€ì&éæM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159119043741696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5005084928133 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164185642840064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722107849123973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43952359 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543177 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81543177 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5066500042752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18527404 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚717102764195840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37590820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159119025129472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚722108206713335 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚725816288823296 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5013082904576 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚720803205918720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37590820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10203311 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43962350 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8324093 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5005425478135 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚725816288823296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18527665 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚725815930437632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012741558272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720803188879360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43962350@4ñ&êëM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37590820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159119025129472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚722108206713335 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚725816288823296 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5013082904576 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚720803205918720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37590820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10203311 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43962350 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8324093 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5005425478135 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚725816288823296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18527665 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚725815930437632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012741558272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720803188879360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43962350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159119043741696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5005084928133 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164185642840064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722107849123973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43952359 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543177 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81543177 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5066500042752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18527404 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚717102764195840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0…w^[ˆ2ë- _wandbruntime‚4046  _runtime‚4046GÍóìÿ 2loss/FBt2/assignment_loss‚0.009968825746643014 /loss/FRt2/detection_loss‚1.3122534503963308 .loss/FBt2/detection_loss‚1.011751262774301 0loss/FRt1/assignment_loss‚1.9256836946204339 2loss/SRqqt1/assignment_loss‚0.5194958307829397 2loss/SRqqt2/assignment_loss‚0.4629398306101776 1loss/FBt1/assignment_loss‚0.02121261628721719 0loss/SRqqt1/detection_loss‚0.987820178621152 0loss/SRqqt2/detection_loss‚0.667055165563824 trainer/global_step‚4749 0loss/FRt2/assignment_loss‚1.0146856047752975 % _timestamp‚1.7842768541787474e+09 _runtime‚ 4046.30407331 %loss/total_loss‚8.986733672056989 epoch‚11 /loss/FRt1/detection_loss‚0.5445725433867421 /loss/FBt1/detection_loss‚0.5092946684919297 _step‚390C8À  í„ /loss/FRt1/detection_loss‚0.5445725433867421 /loss/FBt1/detection_loss‚0.5092946684919297 _step‚390 2loss/FBt2/assignment_loss‚0.009968825746643014 /loss/FRt2/detection_loss‚1.3122534503963308 .loss/FBt2/detection_loss‚1.011751262774301 0loss/FRt1/assignment_loss‚1.9256836946204339 2loss/SRqqt1/assignment_loss‚0.5194958307829397 2loss/SRqqt2/assignment_loss‚0.4629398306101776 1loss/FBt1/assignment_loss‚0.02121261628721719 0loss/SRqqt1/detection_loss‚0.987820178621152 0loss/SRqqt2/detection_loss‚0.667055165563824 trainer/global_step‚4749 0loss/FRt2/assignment_loss‚1.0146856047752975 % _timestamp‚1.7842768541787474e+09 _runtime‚ 4046.30407331 %loss/total_loss‚8.986733672056989 epoch‚11†Ò““î‚(:ˆ ùÎçÒè±ß¤ gpu.0.gpu‚86.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹Ý¬-ï‚(:¢ ùÎçÒ½†§ network.recv‚ 1.092591e+06 disk.sda3.out‚ 134.66015625 disk.dm-1.out‚ 134.66015625 disk.rbd1.out‚93.79443359375 disk.rbd2.in‚0 network.sent‚ 7.422731e+06 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 217.5390625 disk.dm-1.in‚0& disk.nvme0n1.out‚22111.69775390625 disk.sda1.out‚0$ memory_percent‚96.95541858673096" proc.memory.rssMB‚ 2760.8515625) disk./.usagePercent‚36.64948434552841 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0 disk.rbd2.out‚0( proc.memory.availableMB‚ 498.82421875 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 217.5390625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚22111.69775390625 disk.dm-0.in‚ 217.5390625$ disk./.usageGB‚546.1038017272949 disk.sda.out‚ 134.66015625 disk.sda2.out‚0# disk.dm-0.out‚22111.69775390625 disk.sda2.in‚0 disk.rbd3.out‚0* proc.memory.percent‚16.850900650024414 cpu‚26.025643932966968îÞ2ó“ð‚(:ˆ ˆÏçÒ³ÑÕ¤ gpu.0.gpu‚75.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?5 ñ‚(:• ˆÏçÒ¸ÒÆ¥ disk.sda1.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚ 1.097193e+06# disk./.usageGB‚547.423038482666 disk.rbd0.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 217.5390625$ memory_percent‚96.95553779602051 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 cpu‚26.024597027629547 network.sent‚ 7.475033e+06* disk./.usagePercent‚36.738019431094735 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚16.850900650024414! disk.nvme0c0n1.in‚ 217.5390625 disk.sda3.out‚134.9375 disk.rbd1.out‚93.91943359375 disk.sda1.out‚0" proc.memory.rssMB‚ 2760.8515625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚23459.47607421875 disk.dm-1.in‚0 disk.dm-1.out‚134.9375 disk.rbd2.out‚0' proc.memory.availableMB‚ 498.8046875 disk.sda.out‚134.9375 disk.dm-0.in‚ 217.5390625# disk.dm-0.out‚23459.47607421875& disk.nvme0n1.out‚23459.47607421875ðÏëR’ò‚(:‡ —ÏçÒä÷¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`W= ó‚(:• —ÏçÒÁÁò¦ disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd2.in‚0) disk./.usagePercent‚36.64951097031197 disk.rbd1.out‚93.91943359375 disk.drbd2165.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24812.86279296875 disk.sda.out‚135.1875 disk.rbd3.in‚0# disk.dm-0.out‚24812.86279296875& disk.nvme0n1.out‚24812.86279296875" proc.memory.rssMB‚ 2760.8515625 cpu‚26.023338348921126 proc.cpu.threads‚17 disk.nvme0n1.in‚ 218.3671875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0* proc.memory.percent‚16.850900650024414 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚135.1875 disk.rbd2.out‚0 disk.sda3.out‚135.1875 disk.rbd3.out‚0# memory_percent‚96.9557523727417 disk.sda3.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 498.76953125 network.recv‚ 1.100493e+06$ disk./.usageGB‚546.1041984558105 disk.dm-0.in‚ 218.3671875 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 218.3671875 network.sent‚ 7.477955e+06^pw2ô- _wandbruntime‚4085  _runtime‚4085Ó?nOÉõà trainer/global_step‚4799 % _timestamp‚1.7842768936662142e+09 _runtime‚4085.791809045 _step‚391 ' lr-AdamW/pg1‚0.00041631431069388385 ' lr-AdamW/pg2‚0.00041631431069388385nÁþåÎöÈ ' lr-AdamW/pg1‚0.00041631431069388385 ' lr-AdamW/pg2‚0.00041631431069388385 trainer/global_step‚4799 % _timestamp‚1.7842768936662142e+09 _runtime‚4085.791809045 _step‚391‡*îL2÷- _wandbruntime‚4085  _runtime‚4085I“ú:TøÜO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10305148 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165873597750784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5055030911209 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160757052909568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.aÌKŽ on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8409601 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842768936672695e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82372524 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362545 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _step‚392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚728018016370688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44407736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚729336835068137 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚733080782142976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚728017999331328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5062765772288 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5062424433152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165873484270080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5116449972736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5054690361319 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44397759 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5062424433152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5062765772288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44407736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18715016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37964788 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18714746 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚733080782142976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚724281804156928 _runtime‚4085.793023578 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚733080423764480 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8409598 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚160757034297344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5116544841216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5062424433152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165873484270080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5116449972736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5054690361319 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44397759 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5062424433152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5062765772288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44407736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18715016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37964788 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18714746 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚733080782142976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚724281804156928 _runtime‚4085.793023578 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚733080423764480 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8409598 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚160757034297344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5116544841216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165873597750784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5055030911209 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160757052909568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8409601 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842768936672695e+09 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82372524 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82362545 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _step‚392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ˆØï2ú- _wandbruntime‚4086  _runtime‚4086)f½í&ûçM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _step‚393 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚733229035806720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37972420 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _runtime‚4086.406713378 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10307065 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165907923852288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160790463055872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚728165240360960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚728165240360960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37972418 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165908043617280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5063437060096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8411343 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5117460796416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5063437060096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚733228677421056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5117561949184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160790481668096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚729484356793843 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚728165257400320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389222 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18718678 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37972418 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18718408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44416802 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44416802 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724428314828800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5063778406400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44406811 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44406811 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚733229035806720 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37972418 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18718408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44416802 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44416802 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724428314828800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5063778406400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44406811 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44406811 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚733229035806720 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10307065 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165907923852288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160790463055872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚728165240360960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚728165240360960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37972418 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165908043617280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5063437060096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8411343 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5117460796416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5063437060096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚733228677421056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5117561949184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160790481668096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚729484356793843 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚728165257400320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088‰#.h2ý- _wandbruntime‚4086  _runtime‚4086 s>Éþ .loss/FBt1/detection_loss‚0.344300010809996 .loss/FBt2/detection_loss‚0.903101928957871 %loss/total_loss‚8.557158879192272 % _timestamp‚1.7842768942828195e+09 2loss/SRqqt2/assignment_loss‚0.5166635891764837 3loss/FBt2/assignment_loss‚0.0019089167131612938 1loss/SRqqt2/detection_loss‚0.6452474219673338 _step‚394 0loss/FRt1/assignment_loss‚1.8519715759360675 0loss/FRt2/assignment_loss‚0.9932566372306517 /loss/FRt2/detection_loss‚1.2372879061887936 epoch‚11 trainer/global_step‚4799 _runtime‚4086.408145481 2loss/SRqqt1/assignment_loss‚0.5043154526070257 1loss/FBt1/assignment_loss‚0.01825888825547527 /loss/FRt1/detection_loss‚0.5108211172675153 0loss/SRqqt1/detection_loss‚1.030025434081897e:Éä ÿ† 0loss/FRt2/assignment_loss‚0.9932566372306517 /loss/FRt2/detection_loss‚1.2372879061887936 epoch‚11 trainer/global_step‚4799 _runtime‚4086.408145481 2loss/SRqqt1/assignment_loss‚0.5043154526070257 1loss/FBt1/assignment_loss‚0.01825888825547527 /loss/FRt1/detection_loss‚0.5108211172675153 0loss/SRqqt1/detection_loss‚1.030025434081897 .loss/FBt1/detection_loss‚0.344300010809996 .loss/FBt2/detection_loss‚0.903101928957871 %loss/total_loss‚8.557158879192272 % _timestamp‚1.7842768942828195e+09 2loss/SRqqt2/assignment_loss‚0.5166635891764837 3loss/FBt2/assignment_loss‚0.0019089167131612938 1loss/SRqqt2/detection_loss‚0.6452474219673338 _step‚394 0loss/FRt1/assignment_loss‚1.8519715759360675Šjæy“€‚(:ˆ ¦ÏçÒ—•¼¥ gpu.0.gpu‚49.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚290.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.1812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!³ÄÁ(‚(: ¦ÏçÒ®õ⧠disk.drbd2165.in‚0 disk.sda2.in‚0" disk.dm-0.out‚24819.2783203125 disk.sda2.out‚0$ memory_percent‚96.92003726959229 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 219.08203125 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.sda.out‚ 135.4453125 cpu‚26.02994425129309 proc.cpu.threads‚17$ disk./.usageGB‚546.1041221618652 disk.sda3.in‚0 disk.dm-0.in‚ 219.08203125 disk.rbd0.out‚0) disk./.usagePercent‚36.64950585016129 disk.sda3.out‚ 135.4453125 disk.rbd3.out‚0 disk.rbd2.out‚0% disk.nvme0n1.out‚24819.2783203125( proc.memory.availableMB‚ 504.62109375 disk.rbd1.out‚93.98193359375 disk.nvme0n1.in‚ 219.08203125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2766.48046875) proc.memory.percent‚16.88525676727295 network.sent‚ 7.49618e+06 disk.dm-1.out‚ 135.4453125 disk.rbd2.in‚0' disk.nvme0c0n1.out‚24819.2783203125 network.recv‚ 1.103833e+06 disk.dm-1.in‚0 disk.rbd3.in‚0JË>2‚- _wandbruntime‚4103  _runtime‚4103e¶KûÇ=ƒÁ{ ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.20370370149612427 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19258153438568115 *jet/accuracy_1_of_4‚0.9709402918815613 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 _step‚395 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13147583603858948 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4941677749156952 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49076905846595764 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03385278210043907 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8704728484153748 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4677915573120117 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.36574074625968933 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_3_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8965872526168823 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7253142595291138 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7726110816001892 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 particle/accuracy_2_of_2‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23891258239746094 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6585536003112793 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06399454176425934 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7888889312744141 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2315669059753418 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7869426012039185 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7455498576164246 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8005660176277161 *jet/accuracy_2_of_3‚0.5858694911003113 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7149091362953186 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2092021107673645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.6214513778686523 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1491568386554718 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2469145506620407 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9930487275123596 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8483715057373047 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22142232954502106 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.16371700167655945 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8026049733161926 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939215779304504 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30441251397132874 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6964215040206909 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5648350715637207 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025469819083809853 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5208247303962708 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9833964109420776 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24692435562610626 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21117912232875824 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03076385334134102 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2962532341480255 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9832671284675598 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 'particle/f_score‚0.6392362713813782 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7590429782867432 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021273188292980194 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02737831510603428 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7326812744140625 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4128339886665344 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6745327711105347 /particle/accuracy_4_of_4‚0.9366051554679871 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9079388380050659 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7697726488113403 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05149585381150246 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7288151979446411 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7516831159591675 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_4‚0.08627570420503616 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7508253455162048 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +jet/accuracy_3_of_3‚0.07034274190664291 +particle/specificity‚0.9117059111595154 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2281443476676941 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025273194536566734 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780851006507874 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23491910099983215 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1989872306585312 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6302693486213684 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03076385334134102 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.722663402557373 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚4103.510693923 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4107746481895447 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19121649861335754 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18102842569351196 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021123308688402176 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3396093547344208 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22728721797466278 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.20370370149612427 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967526793479919 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6924939751625061 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_4‚1 trainer/global_step‚4799 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973446130752563 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7051670551300049 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19828371703624725 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24753014743328094 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09166666865348816 *jet/accuracy_2_of_4‚0.8006685972213745 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6894286870956421 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04405386000871658 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 particle/accuracy_5_of_5‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8482604026794434 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )jet/accuracy_1_of_1‚0.346746027469635 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2447916716337204 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.98869788646698 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6170877814292908 *jet/accuracy_1_of_2‚0.8060928583145142 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5061362385749817 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.499143159632876 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8483715057373047 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7623295783996582 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21095697581768036 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6567376255989075 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9906672239303589 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9208858013153076 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6852859854698181 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6223794221878052 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5697592496871948 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23604115843772888 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 epoch‚11 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959062933921814 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05566234886646271 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6920543909072876 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8981481194496155 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6253841519355774 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938815832138062 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5333333611488342 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49915841221809387 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6890106201171875 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11536329239606857 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7758504152297974 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22772091627120972 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7179080843925476 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7172147035598755 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_3‚0.9303398132324219 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7845945954322815 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7596768140792847 (particle/accuracy‚0.8493730425834656 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5807291269302368 % _timestamp‚1.7842769113844967e+09 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7674340605735779 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7118641138076782 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6924939751625061 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23604115843772888 6validation_average_jet_accuracy‚0.3881016671657562 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7693009972572327 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24715307354927063 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7918179035186768 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5891454815864563 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7784781455993652 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949609637260437 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6543461680412292 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13249365985393524 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3164285123348236 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1·úò%„Æ{ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939215779304504 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30441251397132874 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6964215040206909 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5648350715637207 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025469819083809853 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5208247303962708 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9833964109420776 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24692435562610626 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21117912232875824 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03076385334134102 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2962532341480255 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9832671284675598 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 'particle/f_score‚0.6392362713813782 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7590429782867432 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021273188292980194 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02737831510603428 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7326812744140625 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4128339886665344 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6745327711105347 /particle/accuracy_4_of_4‚0.9366051554679871 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9079388380050659 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7697726488113403 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05149585381150246 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7288151979446411 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7516831159591675 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_4‚0.08627570420503616 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7508253455162048 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +jet/accuracy_3_of_3‚0.07034274190664291 +particle/specificity‚0.9117059111595154 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2281443476676941 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025273194536566734 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780851006507874 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23491910099983215 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1989872306585312 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6302693486213684 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03076385334134102 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.722663402557373 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚4103.510693923 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4107746481895447 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19121649861335754 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18102842569351196 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021123308688402176 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3396093547344208 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22728721797466278 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.20370370149612427 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9967526793479919 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6924939751625061 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_4‚1 trainer/global_step‚4799 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973446130752563 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7051670551300049 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19828371703624725 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24753014743328094 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09166666865348816 *jet/accuracy_2_of_4‚0.8006685972213745 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6894286870956421 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04405386000871658 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 particle/accuracy_5_of_5‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8482604026794434 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )jet/accuracy_1_of_1‚0.346746027469635 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2447916716337204 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.98869788646698 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6170877814292908 *jet/accuracy_1_of_2‚0.8060928583145142 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5061362385749817 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.499143159632876 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8483715057373047 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7623295783996582 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21095697581768036 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6567376255989075 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9906672239303589 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9208858013153076 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6852859854698181 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 0"Purity/0FRt2Smov­Rqqt1FBt/SRqqt_purity‚ 0.3984375 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6223794221878052 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5697592496871948 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23604115843772888 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 epoch‚11 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959062933921814 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05566234886646271 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6920543909072876 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8981481194496155 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6253841519355774 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938815832138062 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5333333611488342 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49915841221809387 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6890106201171875 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11536329239606857 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7758504152297974 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22772091627120972 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7179080843925476 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7172147035598755 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_3‚0.9303398132324219 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7845945954322815 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7596768140792847 (particle/accuracy‚0.8493730425834656 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5807291269302368 % _timestamp‚1.7842769113844967e+09 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7674340605735779 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7118641138076782 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6924939751625061 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23604115843772888 6validation_average_jet_accuracy‚0.3881016671657562 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7693009972572327 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24715307354927063 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7918179035186768 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5891454815864563 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7784781455993652 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949609637260437 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6543461680412292 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13249365985393524 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3164285123348236 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.20370370149612427 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19258153438568115 *jet/accuracy_1_of_4‚0.9709402918815613 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 _step‚395 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13147583603858948 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4941677749156952 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49076905846595764 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03385278210043907 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8704728484153748 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4677915573120117 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.36574074625968933 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_3_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8965872526168823 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7253142595291138 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7726110816001892 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 particle/accuracy_2_of_2‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23891258239746094 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6585536003112793 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06399454176425934 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7888889312744141 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2315669059753418 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7869426012039185 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7455498576164246 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8005660176277161 *jet/accuracy_2_of_3‚0.5858694911003113 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7149091362953186 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2092021107673645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.6214513778686523 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1491568386554718 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2469145506620407 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9930487275123596 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8483715057373047 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22142232954502106 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.16371700167655945 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8026049733161926‹zäÚž…‚(:“ µÏçÒ “Æ¢ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùã,Î#†‚(:˜ µÏçÒ“¡­¦ disk.dm-1.in‚0 disk.dm-1.out‚ 135.6171875 disk.rbd1.in‚ 0.01171875! disk.dm-0.out‚25074.544921875 proc.cpu.threads‚17 disk.sda2.out‚0& disk.nvme0c0n1.out‚25074.544921875 disk.sda.out‚ 135.6171875 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 219.91015625$ memory_percent‚95.45779228210449 disk.rbd0.out‚0 disk.rbd3.out‚0 network.sent‚ 7.55909e+06* disk./.usagePercent‚36.649306676299645$ disk./.usageGB‚546.1011543273926 disk.sda3.out‚ 135.6171875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 744.1953125 disk.sda3.in‚0 disk.rbd1.out‚93.98193359375 disk.nvme0n1.in‚ 219.91015625 disk.sda.in‚0( proc.memory.percent‚16.9863224029541 disk.sda2.in‚0 disk.dm-0.in‚ 219.91015625$ disk.nvme0n1.out‚25074.544921875 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2783.0390625 network.recv‚ 1.108511e+06 disk.rbd2.out‚0 disk.drbd2165.out‚0 cpu‚26.03647265616618P6/¤ž‡‚(:“ ÄÏçÒ³×ô¤ gpu.0.gpu‚43.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡_ûýˆ‚(:Œ ÄÏçÒ®·ï§% disk.nvme0n1.out‚25327.4384765625' disk.nvme0c0n1.out‚25327.4384765625 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2795.04296875 disk.rbd0.out‚0) proc.memory.percent‚17.05958843231201 disk.sda.in‚0 disk.sda.out‚ 135.890625 disk.rbd1.out‚94.04443359375 disk.sda3.in‚0 disk.sda3.out‚ 135.890625 disk.rbd2.in‚0 disk.nvme0n1.in‚220.625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.43373584747314) disk./.usagePercent‚36.64937016616815$ disk./.usageGB‚546.1021003723145 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd2.out‚0 network.sent‚ 7.605802e+06 network.recv‚ 1.112643e+06( proc.memory.availableMB‚ 748.13671875 cpu‚26.03518261686261 disk.rbd0.in‚0 disk.nvme0c0n1.in‚220.625 disk.sda2.in‚0 disk.dm-1.out‚ 135.890625 disk.dm-0.in‚220.625 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0" disk.dm-0.out‚25327.4384765625Æ㞉‚(:“ ÓÏçÒó÷‹¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l`+ý!Š‚(:– ÓÏçÒ°ùƒ§ disk.sda1.out‚0# proc.memory.rssMB‚ 2805.97265625 cpu‚26.034051996300636# disk.dm-0.out‚25333.27197265625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚220.625 disk.sda3.out‚ 136.33203125 disk.rbd3.in‚0 network.recv‚ 1.116073e+06* disk./.usagePercent‚36.649401143079785 disk.dm-0.in‚220.625 disk.rbd0.out‚0 disk.nvme0n1.in‚220.625 disk.sda.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 744.9453125 proc.cpu.threads‚17 disk.rbd2.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚94.04443359375$ memory_percent‚95.45321464538574 network.sent‚ 7.610854e+06$ disk./.usageGB‚546.1025619506836( disk.nvme0c0n1.out‚25333.27197265625 disk.rbd3.out‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚25333.27197265625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.12629795074463 disk.sda.out‚ 136.33203125 disk.sda3.in‚0 disk.dm-1.out‚ 136.33203125È·Á2‹- _wandbruntime‚4144  _runtime‚4144"ü©ÉŒÃ ' lr-AdamW/pg1‚0.00041508100160966166 ' lr-AdamW/pg2‚0.00041508100160966166 trainer/global_step‚4849 % _timestamp‚1.7842769526674452e+09 _runtime‚4144.793040036 _step‚396FbV¬ÎÈ _runtime‚4144.793040036 _step‚396 ' lr-AdamW/pg1‚0.00041508100160966166 ' lr-AdamW/pg2‚0.00041508100160966166 trainer/global_step‚4849 % _timestamp‚1.7842769526674452e+09ŒT.jÆ2Ž- _wandbruntime‚4144  _runtime‚4144 Ömìá'ÛO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733515983925248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38467054 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38467052 _step‚397 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842769526684844e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5131194482176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44932515 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162444402935808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8499371 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5185928316928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5130853143040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5130853143040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167630232452096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5185848128512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38467052 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742440470606336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83399569 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5130853143040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5130853143040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167630232452096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segÜEfNÿments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5185848128512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38467052 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742440470606336U8÷2‘- _wandbruntime‚4145  _runtime‚4145Q ¸Ïë&’åM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚733662511636480 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44941581 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚738786889813665 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416267 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38474686 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8501113 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167664793216512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚737456875532288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5124037627183 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38474684 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44931590 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38474686 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8501116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10443015 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5186885535232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406274 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38474684 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44941581 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737456875532288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742589082648576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842769532826314e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18944389 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162477813082112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5186961522176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚398 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167664698617344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44931590 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737456858492928 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742589082648576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742588724262912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚738786532224303 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406274 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18944128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5132207116288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416267 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18944128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5132207116288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416267 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44941581 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚738786889813665 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416267 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38474686 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8501113 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167664793216512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚737456875532288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5124037627183 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38474684 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44931590 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38474686 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8501116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10443015 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5186885535232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406274 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38474684 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44941581 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737456875532288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742589082648576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842769532826314e+09Ž€Ì“(2”- _wandbruntime‚4145  _runtime‚4145š+0 •ƒ 0loss/SRqqt1/detection_loss‚1.017780191234636 _runtime‚4145.409822506 /loss/FRt1/assignment_loss‚1.825519354995765 2loss/SRqqt2/assignment_loss‚0.5010776921655168 /loss/FRt2/detection_loss‚1.2548255473179541 1loss/SRqqt2/detection_loss‚0.6448367022991152 0loss/FRt2/assignment_loss‚1.0508710457446562 2loss/FBt1/assignment_loss‚0.011230020282089607 /loss/FRt1/detection_loss‚0.5078230145227359 /loss/FBt2/detection_loss‚0.8120424345796627 epoch‚12 _step‚399 2loss/SRqqt1/assignment_loss‚0.5200937459715738 /loss/FBt1/detection_loss‚0.4229792534807536 %loss/total_loss‚8.570995993008463 trainer/global_step‚4849 % _timestamp‚1.7842769532845006e+09 3loss/FBt2/assignment_loss‚0.0019169904140055897ãÓÄ–ˆ epoch‚12 _step‚399 2loss/SRqqt1/assignment_loss‚0.5200937459715738 /loss/FBt1/detection_loss‚0.4229792534807536 %loss/total_loss‚8.570995993008463 trainer/global_step‚4849 % _timestamp‚1.7842769532845006e+09 3loss/FBt2/assignment_loss‚0.0019169904140055897 0loss/SRqqt1/detection_loss‚1.017780191234636 _runtime‚4145.409822506 /loss/FRt1/assignment_loss‚1.825519354995765 2loss/SRqqt2/assignment_loss‚0.5010776921655168 /loss/FRt2/detection_loss‚1.2548255473179541 1loss/SRqqt2/detection_loss‚0.6448367022991152 0loss/FRt2/assignment_loss‚1.0508710457446562 2loss/FBt1/assignment_loss‚0.011230020282089607 /loss/FRt1/detection_loss‚0.5078230145227359 /loss/FBt2/detection_loss‚0.8120424345796627›Jáà“—‚(:ˆ âÏçÒÊóª¤ gpu.0.gpu‚90.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚235.703' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.1406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z¼ÿ§˜‚(:’ âÏçÒ„Ë‹¦ disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda3.out‚136.4375' proc.memory.availableMB‚ 742.5234375 proc.cpu.threads‚17 network.recv‚ 1.119348e+06# disk.dm-0.out‚25340.56103515625 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875$ memory_percent‚95.46799659729004 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚136.4375 disk.dm-0.in‚ 221.453125 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 221.453125( disk.nvme0c0n1.out‚25340.56103515625 disk.sda.out‚136.4375 cpu‚26.03293100914694 disk.nvme0n1.in‚ 221.453125 disk.sda3.in‚0 disk.rbd1.out‚94.10693359375 network.sent‚ 7.626817e+06 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk./.usagePercent‚36.64941266341882$ disk./.usageGB‚546.1027336120605 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd2.out‚0& disk.nvme0n1.out‚25340.56103515625 disk.sda1.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2808.60546875* proc.memory.percent‚17.142367362976074L;÷ê“™‚(:ˆ ñÏçÒÈ‘¤ gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=œ%š‚(:š ñÏçÒÕ·¦$ disk./.usageGB‚546.1069145202637 disk.sda1.out‚0 disk.sda.out‚ 136.6328125 disk.rbd1.in‚ 0.01171875( proc.memory.availableMB‚ 742.51171875 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.out‚ 136.6328125 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 222.16796875 disk.dm-1.out‚ 136.6328125 disk.dm-0.in‚ 222.16796875 disk.nvme0n1.in‚ 222.16796875 cpu‚26.031801061633868 network.sent‚ 7.676805e+06 disk.rbd2.out‚0# disk.nvme0n1.out‚25349.96484375) disk./.usagePercent‚36.64969324767635 disk.sda1.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚25349.96484375 disk.sda2.in‚0# proc.memory.rssMB‚ 2808.60546875 disk.rbd0.out‚0* proc.memory.percent‚17.142367362976074 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚94.10693359375 network.recv‚ 1.123903e+06 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.dm-0.out‚25349.96484375 disk.rbd2.in‚0$ memory_percent‚95.46806812286377t)À2›- _wandbruntime‚4184  _runtime‚4184%8JÆœÀ trainer/global_step‚4899 $ _timestamp‚1.784276992654471e+09 _runtime‚4184.781447396 _step‚400 & lr-AdamW/pg1‚0.0004138362119464121 & lr-AdamW/pg2‚0.0004138362119464121š8T¹ËÅ & lr-AdamW/pg1‚0.0004138362119464121 & lr-AdamW/pg2‚0.0004138362119464121 trainer/global_step‚4899 $ _timestamp‚1.784276992654471e+09 _runtime‚4184.781447396 _step‚400è°kÅ2ž- _wandbruntime‚4184  _runtime‚4184ÈoýºÞŸÛO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45386967 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚744671668944896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181548644864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164115840862208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚401 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225642 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38848654 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181548644864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45376990 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181889984000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5173983610259 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚749853217589760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19131333 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚4184.781547895 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚749853217589760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38848654 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45376990 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚744671685984256 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚749853575968256 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744671685984256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744671668944896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164115822249984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19131590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚740841534558208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842769926552353e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚4899 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38848652 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8586621 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169353071403008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5237325147136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10544969 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5237249153024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.alß©Í located‚45386967 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚746015160579217 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38848652 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚749853575968256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169353166009344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚740841517518848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5173643060369 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5181889984000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84235621 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10544715 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8586618 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10544715 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8586618 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5173983610259 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚749853217589760 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19131333 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚4184.781547895 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚749853217589760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38848654 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45376990 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚744671685984256 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚749853575968256 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744671685984256 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744671668944896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164115822249984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19131590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚740841534558208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842769926552353e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚4899 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38848652 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8586621 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169353071403008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5237325147136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10544969 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5237249153024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269‘G˜1î“¡‚(:ˆ €ÐçÒúó›¤ gpu.0.gpu‚65.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÜ-¢‚(:¢ €ÐçÒ‰”÷¦$ disk./.usageGB‚546.1069374084473 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 136.85546875) disk./.usagePercent‚36.64969478372156 disk.rbd1.out‚94.16943359375 disk.nvme0n1.in‚ 222.99609375( proc.memory.availableMB‚ 742.26953125* proc.memory.percent‚17.142367362976074 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 222.99609375 disk.sda.out‚ 136.85546875 disk.rbd1.in‚ 0.01171875 disk.rbd3.out‚0" disk.dm-0.out‚25356.1279296875 disk.drbd2165.out‚0 network.recv‚ 1.126522e+06 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 222.99609375% disk.nvme0n1.out‚25356.1279296875 disk.sda1.out‚0# memory_percent‚95.4695463180542# proc.memory.rssMB‚ 2808.60546875' disk.nvme0c0n1.out‚25356.1279296875 disk.sda.in‚0 disk.sda3.out‚ 136.85546875 disk.rbd2.out‚0 cpu‚26.030677376038533 network.sent‚ 7.681682e+06 disk.drbd2165.in‚0|-öÛ2£- _wandbruntime‚4185  _runtime‚4185â4lEë&¤åM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5182561271808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚740988028190720 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚750001471246336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10546783 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744818909974528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252319 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45386042 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38856284 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242326 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744818927013888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174654114091 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750001829632000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19135146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5238263095296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5174994664093 _step‚402 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8588363 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5182561271808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169387621285888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚ 4185.39829022 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38856284 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10547055 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38856286 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45396033 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5182902618112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚744818927013888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164149251008512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5182902618112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45396033 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164149269620736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚740988045230080ŸK ¥êM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169387514103808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚746163039894173 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38856284 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242326 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744818927013888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174654114091 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750001829632000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19135146 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5238263095296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5174994664093 _step‚402 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8588363 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5182561271808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169387621285888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚ 4185.39829022 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38856284 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10547055 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38856286 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45396033 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5182902618112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚744818927013888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164149251008512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5182902618112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45396033 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164149269620736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚740988045230080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚K°Ñaç 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5182561271808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚740988028190720 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚750001471246336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10546783 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744818909974528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252319 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45386042 Bchž«‚(:“ žÐçÒÈŽ¥ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ÉcW(¬‚(: žÐçÒÃŽ›§ network.sent‚ 7.787765e+06 disk.sda1.out‚0' disk.nvme0c0n1.out‚25368.6220703125 proc.cpu.threads‚17$ disk./.usageGB‚546.1052856445312 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 223.7109375% disk.nvme0n1.out‚25368.6220703125 disk.drbd2165.out‚0 disk.sda.in‚0$ memory_percent‚96.97000980377197" disk.dm-0.out‚25368.6220703125 disk.rbd2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 496.43359375 network.recv‚ 1.134979e+06) disk./.usagePercent‚36.64958393245924 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 137.3359375 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚94.23193359375 disk.sda2.in‚0 cpu‚26.028403927497425 disk.dm-0.in‚ 223.7109375 disk.rbd2.out‚0 disk.sda.out‚ 137.3359375* proc.memory.percent‚17.142367362976074 disk.sda3.out‚ 137.3359375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 223.7109375# proc.memory.rssMB‚ 2808.60546875ÂV¤ý2­- _wandbruntime‚4224  _runtime‚4224êl³É®Ã _step‚404 ' lr-AdamW/pg1‚0.00041258002165464917 ' lr-AdamW/pg2‚0.00041258002165464917 trainer/global_step‚4949 % _timestamp‚1.7842770326258152e+09 _runtime‚4224.751428589ü!†§Î¯È ' lr-AdamW/pg1‚0.00041258002165464917 ' lr-AdamW/pg2‚0.00041258002165464917 trainer/global_step‚4949 % _timestamp‚1.7842770326258152e+09 _runtime‚4224.751428589 _step‚404”=~W2°- _wandbruntime‚4224  _runtime‚4224¸µâ'±ÜO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757265964573184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚748167051112448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4949 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚757265964573184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5232244146688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071673 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753391668248975 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19323558 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5232585485824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8673871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165787260176384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚752033737465856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171075932938752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚748167068151808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚757266322951680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45831442 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171076035933696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10649454 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8673868 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5232585485824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45841419 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19323322 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5288757145088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39230252 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45841419 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚752033737465856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5232244146688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224600097167 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚405 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752033720426496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10649687 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757265964573184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚748167051112448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4949 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚757265964573184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5232244146688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071673 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753391668248975 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19323558 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5232585485824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8673871 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165787260176384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚752033737465856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171075932938752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚748167068151808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚757266322951680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45831442 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171076035933696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10649454 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8673868 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5232585485824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45841419 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438•™®C2³- _wandbruntime‚4225  _runtime‚4225Ñç”^ì´åM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45850485 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5233256773632 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚748313578823680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10651714 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45850485 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45840494 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚753538832385319 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚4225.365937812 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5233256773632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752180978495488 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8675616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚406 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8675613 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19327062 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225270600999 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_g¤,ÿbytes.small_pool.freed‚ 5289685336576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784277033240085e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5289784392192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚757414576615424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19327330 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078378 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088371 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171110491939328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10651449 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165820688934912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39237884 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39237886 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚748313561784320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225611151001 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚752180961456128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85078378 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753539189974681 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165820707547136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169øñ‰ð&µêM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171110374271488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088371 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5233598119936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5233598119936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39237886 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39237884 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚748313578823680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10651714 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45850485 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45840494 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚753538832385319 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚4225.365937812 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5233256773632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752180978495488 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8675616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚406 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8675613 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19327062 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225270600999 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5289685336576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784277033240085e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5289784392192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚757414576615424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19327330 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078378 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088371 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171110491939328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10651449 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165820688934912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39237884 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39237886 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚748313561784320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225611151001 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚752180961456128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85078378 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753539189974681 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165820707547136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4949 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45850485 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5233256773632 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167458698102784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8761121 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8761118 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897746 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85907725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46295871 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5340001691136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39611854 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764678711556608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764679069935104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5340077685248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10754314 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚409 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5283280987648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5283280987648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46285894 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10754054 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764678711556608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39611852 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275216584075 $ _timestamp‚1.784277072415372e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759395788947456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5282939648512 _runtime‚ 4264.54114061 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759395771908096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5282939648512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚755492601745408 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚759395771908096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46285894 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19515435 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172798699793920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167458716715008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19515172 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759395788947456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39611854 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚755492584706048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚760767818329483 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5274876034185¸žû¾HÄÞO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85907725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46295871 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5340001691136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39611854 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764678711556608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764679069935104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5340077685248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10754314 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚409 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5283280987648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5283280987648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46285894 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10754054 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764678711556608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39611852 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275216584075 $ _timestamp‚1.784277072415372e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759395788947456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5282939648512 _runtime‚ 4264.54114061 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759395771908096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5282939648512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚755492601745408 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚759395771908096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46285894 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19515435 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172798699793920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167458716715008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19515172 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759395788947456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39611854 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚755492584706048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚760767818329483 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5274876034185 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167458698102784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8761121 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8761118 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897746™?4ÔÝ2Å- _wandbruntime‚4265  _runtime‚4265¹wÜì&ÆæM B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚760914982465827 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19518988 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759543029977088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759543012937728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10756125 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚4265.154492739 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764827323598848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5275887087907 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283952275456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914430 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284293621760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46294946 % _timestamp‚1.7842770730286345e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46294946 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924423 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39619484 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762866 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚755639112417280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚759543029977088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284293621760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5283952275456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3Ï®¹ñ&ÇëM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19518988 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759543029977088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759543012937728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10756125 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚4265.154492739 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚764827323598848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5275887087907 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283952275456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914430 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284293621760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46294946 % _timestamp‚1.7842770730286345e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46294946 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924423 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39619484 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762866 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚755639112417280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚759543029977088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284293621760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5283952275456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚760914982465827 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768143968409991 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46740346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5333976489472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10859288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169130136029184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71799808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766757823389696 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174521532250112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46750323 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5325492521093 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39993452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19707404 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚762818135339008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39993454 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772091458540032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚772091458540032 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743777 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46740346 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733798 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772091816918528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚772091816918528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚766757840429056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8848371 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚762818135339008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39993454 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772091458540032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚772091458540032 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743777 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46740346 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733798 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772091816918528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚772091816918528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚766757840429056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8848371 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768143968409991 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46740346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5333976489472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10859288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169130136029184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71799808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766757823389696 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174521532250112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46750323 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5325492521093 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39993452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19707404 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46749398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10861111 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762964646010880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5392406590976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8850116 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326503574815 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚766905064419328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40001086 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19711224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86760475 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86750482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40001086 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚768291132546335 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760475 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174555882808832 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766905081458688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19711489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10861373 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5326844124817 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚772239712196608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46759389 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46749398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10861111 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762964646010880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5392406590976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8850116 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326503574815 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚766905064419328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40001086 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19711224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86760475 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86750482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40001086 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚768291132546335 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760475 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174555882808832 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚766905081458688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19711489 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10861373 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5326844124817 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚772239712196608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46759389 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170801573955584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47204775 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _step‚417 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779504205523456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40375054 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47194798 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5384330652160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5099 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774119891910656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5384671991296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47204775 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579829 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10963528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5442639654912 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19898887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5376449557891 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40375052 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Dé(ˆjæàO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5376449557891 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40375052 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170801592567808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775520118490499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176244213610496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770143668932608 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5384671991296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5442713551872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170801573955584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47204775 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _step‚417 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779504205523456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ ¶LA| 40375054 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47194798 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5384330652160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5099 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774119891910656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5384671991296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47204775 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579829 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10963528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5442639654912 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19898887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0¡;´c 2ç- _wandbruntime‚4344  _runtime‚4344Éù=³ì&èæM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10965350 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚770290162565120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40382686 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19902976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5385684625408 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596527 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47203850 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774267132940288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5385684625408 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40382686 _runtime‚4344.995663052 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596527 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19902713 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5385343279104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚774267132940288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775667282626843 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775667640216205 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596527 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47203850 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚774267132940288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5385684625408 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40382686 _runtime‚4344.995663052 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596527 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19902713 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5385343279104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚774267132940288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚775667282626843 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚775667640216205 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19902976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5385684625408 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576¢¹@/^2ê- _wandbruntime‚4344  _runtime‚4344®_û± ë„ trainer/global_step‚5099 1loss/SRqqt2/detection_loss‚0.6476808754401757 /loss/FBt1/detection_loss‚0.5039779247526356 .loss/FBt2/detection_loss‚1.055120043234811 2loss/SRqqt2/assignment_loss‚0.5286809613312586 1loss/SRqqt1/detection_loss‚1.0300969499769772 0loss/FRt1/assignment_loss‚1.8438011410489548 /loss/FRt1/detection_loss‚0.5074406234983595 /loss/FRt2/detection_loss‚1.2518488606813438 % _timestamp‚1.7842771528715725e+09 _runtime‚4344.996889857 _step‚419 0loss/FRt2/assignment_loss‚0.9020230817622032 2loss/SRqqt1/assignment_loss‚0.5387346266043308 2loss/FBt1/assignment_loss‚0.006102220847947717 3loss/FBt2/assignment_loss‚0.0031380449947305137 %loss/total_loss‚8.818645354173729 epoch‚12Ó¡hì‰ _step‚419 0loss/FRt2/assignment_loss‚0.9020230817622032 2loss/SRqqt1/assignment_loss‚0.5387346266043308 2loss/FBt1/assignment_loss‚0.006102220847947717 3loss/FBt2/assignment_loss‚0.0031380449947305137 %loss/total_loss‚8.818645354173729 epoch‚12 trainer/global_step‚5099 1loss/SRqqt2/detection_loss‚0.6476808754401757 /loss/FBt1/detection_loss‚0.5039779247526356 .loss/FBt2/detection_loss‚1.055120043234811 2loss/SRqqt2/assignment_loss‚0.5286809613312586 1loss/SRqqt1/detection_loss‚1.0300969499769772 0loss/FRt1/assignment_loss‚1.8438011410489548 /loss/FRt1/detection_loss‚0.5074406234983595 /loss/FRt2/detection_loss‚1.2518488606813438 % _timestamp‚1.7842771528715725e+09 _runtime‚4344.996889857£FΞí‚(:“ ¥ÑçÒæ禧 gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚112.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.461799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'o'î‚(:œ ¥ÑçÒ…ÞŒ¬ network.recv‚ 1.169826e+06 disk.nvme0n1.in‚ 229.21875 disk.sda1.in‚0$ memory_percent‚96.97668552398682# proc.memory.rssMB‚ 2808.60546875) disk./.usagePercent‚36.64929336390787 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 229.21875 disk.sda1.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.142367362976074 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 142.36328125 disk.rbd3.in‚0 disk.rbd0.out‚0 network.sent‚ 8.04204e+06 disk.sda2.out‚0 disk.sda3.out‚ 142.36328125 disk.rbd2.out‚0 disk.rbd0.in‚0$ disk./.usageGB‚546.1009559631348( disk.nvme0c0n1.out‚25444.72314453125 disk.sda.in‚0 disk.nvme0c0n1.in‚ 229.21875 disk.sda3.in‚0 disk.rbd1.out‚94.48193359375 disk.rbd3.out‚0 disk.dm-1.out‚ 142.36328125# disk.dm-0.out‚25444.72314453125& disk.nvme0n1.out‚25444.72314453125( proc.memory.availableMB‚ 495.33984375 cpu‚26.018882853227943¼c|J“ï‚(:ˆ ´ÑçÒªÊä gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ƒþ,ð‚(:¡ ´ÑçÒ©Ïü¦# proc.memory.rssMB‚ 2808.60546875* proc.memory.percent‚17.142367362976074 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 229.93359375 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 229.93359375( disk./.usagePercent‚36.6494244397654$ disk./.usageGB‚546.1029090881348 disk.dm-1.out‚ 142.67578125 disk.rbd1.out‚94.54443359375 disk.nvme0n1.in‚ 229.93359375% disk.nvme0n1.out‚25451.7333984375( proc.memory.availableMB‚ 495.03515625 network.sent‚ 8.107206e+06 disk.sda1.out‚0 disk.sda.out‚ 142.67578125 cpu‚26.017748242619966" disk.dm-0.out‚25451.7333984375 disk.sda2.in‚0 disk.rbd0.out‚0 network.recv‚ 1.17404e+06 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚96.97854518890381 disk.sda3.out‚ 142.67578125 disk.rbd2.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚25451.7333984375 disk.rbd2.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0D{a<’ñ‚(:‡ ÃÑçÒþŒ°¤ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚254.805' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËkU-$ò‚(:™ ÃÑçÒÃÇÝ¥# disk.nvme0n1.out‚25462.92578125$ memory_percent‚96.97701930999756 disk.sda.out‚ 142.9921875 disk.nvme0n1.in‚ 230.76171875 disk.sda1.in‚0( proc.memory.availableMB‚ 495.28515625 disk.sda2.out‚0 disk.dm-0.in‚ 230.76171875 disk.rbd2.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 142.9921875 disk.sda.in‚0 disk.rbd1.out‚94.54443359375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2808.60546875* proc.memory.percent‚17.142367362976074 disk.dm-1.out‚ 142.9921875) disk./.usagePercent‚36.64941803957704" disk.nvme0c0n1.in‚ 230.76171875 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 cpu‚26.01674300123339% disk.nvme0c0n1.out‚25462.92578125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 network.sent‚ 8.112266e+06 network.recv‚ 1.177255e+06$ disk./.usageGB‚546.1028137207031 disk.dm-0.out‚25462.92578125 disk.drbd2165.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0œ‘s2ó- _wandbruntime‚4384  _runtime‚4384»*3ÇôÁ _step‚420 & lr-AdamW/pg1‚0.0004074428787320307 & lr-AdamW/pg2‚0.0004074428787320307 trainer/global_step‚5149 % _timestamp‚1.7842771922058918e+09 _runtime‚4384.331485964Š<Ò[ÌõÆ & lr-AdamW/pg2‚0.0004074428787320307 trainer/global_step‚5149 % _timestamp‚1.7842771922058918e+09 _runtime‚4384.331485964 _step‚420 & lr-AdamW/pg1‚0.0004074428787320307¤”:¬2ö- _wandbruntime‚4384  _runtime‚4384g¢àQ{ ÷ÛO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚781481926352896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88415881 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177966884739072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172473030494208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11067899 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405902 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47649250 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5435367493120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5435026153984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚5149 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427066044799 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚781481926352896 QI¼Nd fDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177966777549824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40756654 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47659227 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5493765667840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782895910981757 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40756654 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47659227 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5426725494909 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786916952506880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777469202526208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777469185486848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5435026153984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5493854244864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚782896268571007 % _timestamp‚1.7842771922070236e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚781481943392256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20090767 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786916952506880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _runtime‚ 4384.33280796 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786917310885376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚421 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚786917310885376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9022871 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40756652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11068159 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40756652 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435367493120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47649250 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚781481943392256 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169Á#RÛæ'øàO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786916952506880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _runtime‚ 4384.33280796 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786917310885376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚421 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚786917310885376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9022871 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40756652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11068159 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40756652 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435367493120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47649250 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚781481943392256 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚781481926352896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88415881 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177966884739072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172473030494208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11067899 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405902 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47649250 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5435367493120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5435026153984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚5149 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427066044799 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚781481926352896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177966777549824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40756654 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47659227 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5493765667840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782895910981757 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40756654 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47659227 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5426725494909 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786916952506880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777469202526208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777469185486848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5435026153984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5493854244864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚782896268571007 % _timestamp‚1.7842771922070236e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚781481943392256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20090767 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976¥q©·2ù- _wandbruntime‚4384  _runtime‚4384tÓóì&úæM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47668293 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚781629184421888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚781629184421888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚781629167382528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777615713198080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20094554 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783043432707351 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40764284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5427736548631 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9024613 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172506440640512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5436038780928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432579 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428077098633 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5436038780928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚781629167382528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40764286 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178001221414400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88432579 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422586 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47658302 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚1†ô+~2154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422586 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47658302 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47668293 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚781629184421888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚781629184421888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚781629167382528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚777615713198080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20094554 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783043432707351 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40764284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5427736548631 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9024613 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172506440640512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5436038780928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432579 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428077098633 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5436038780928¦í0È2ü- _wandbruntime‚4384  _runtime‚4384@©’ý 0loss/FRt2/assignment_loss‚1.0060114797966255 2loss/SRqqt1/assignment_loss‚0.5434028252384603 2loss/FBt1/assignment_loss‚0.024908823985591338 epoch‚12 _step‚423 /loss/FRt1/assignment_loss‚1.883401186737631 2loss/FBt2/assignment_loss‚0.008120548393587038 /loss/FBt1/detection_loss‚0.5151475837735202 /loss/FBt2/detection_loss‚0.7974871282155315 %loss/total_loss‚9.038195869830528 trainer/global_step‚5149 /loss/FRt1/detection_loss‚0.5738546501001878 0loss/SRqqt2/detection_loss‚0.644082262009126 1loss/SRqqt2/assignment_loss‚0.597100287553983 /loss/FRt2/detection_loss‚1.4109126300116637 1loss/SRqqt1/detection_loss‚1.0337664640146194 % _timestamp‚1.7842771928226821e+09 _runtime‚4384.948004834Èd:È þ† /loss/FBt1/detection_loss‚0.5151475837735202 /loss/FBt2/detection_loss‚0.7974871282155315 %loss/total_loss‚9.038195869830528 trainer/global_step‚5149 /loss/FRt1/detection_loss‚0.5738546501001878 0loss/SRqqt2/detection_loss‚0.644082262009126 1loss/SRqqt2/assignment_loss‚0.597100287553983 /loss/FRt2/detection_loss‚1.4109126300116637 1loss/SRqqt1/detection_loss‚1.0337664640146194 % _timestamp‚1.7842771928226821e+09 _runtime‚4384.948004834 0loss/FRt2/assignment_loss‚1.0060114797966255 2loss/SRqqt1/assignment_loss‚0.5434028252384603 2loss/FBt1/assignment_loss‚0.024908823985591338 epoch‚12 _step‚423 /loss/FRt1/assignment_loss‚1.883401186737631 2loss/FBt2/assignment_loss‚0.008120548393587038§Y©T.œÿ‚(:‘ ÒÑçÒȉ¥ gpu.0.gpu‚15.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.603' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.12060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x˜t«-€‚(:¢ ÒÑçÒðó¯§ disk.sda2.in‚0 disk.dm-1.out‚ 143.12109375* proc.memory.percent‚17.142367362976074# disk.dm-0.out‚25470.10107421875 disk.rbd2.in‚0 disk.rbd2.out‚0 network.sent‚ 8.115188e+06) disk./.usagePercent‚36.64942392775033 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 231.4765625 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda3.out‚ 143.12109375 disk.nvme0n1.in‚ 231.4765625 disk.sda1.in‚0 cpu‚26.01582198919209 disk.rbd0.in‚0( disk.nvme0c0n1.out‚25470.10107421875 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd3.out‚0& disk.nvme0n1.out‚25470.10107421875 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0$ memory_percent‚96.97844982147217( proc.memory.availableMB‚ 495.05078125# proc.memory.rssMB‚ 2808.60546875$ disk./.usageGB‚546.1029014587402 disk.rbd1.out‚94.66943359375 disk.dm-0.in‚ 231.4765625 disk.sda1.out‚0 network.recv‚ 1.180424e+06 disk.sda.out‚ 143.12109375:0L‘‚(:† áÑçÒø‚ð¤ gpu.0.gpu‚39.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ÍDc/‚‚(:¤ áÑçÒÝÇ£§ disk.dm-0.in‚ 232.3046875# disk.dm-0.out‚25479.14697265625 cpu‚26.014815913552425 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 143.35546875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 232.3046875& disk.nvme0n1.out‚25479.14697265625 network.sent‚ 8.180305e+06$ disk./.usageGB‚546.1026954650879 disk.rbd0.in‚0( disk.nvme0c0n1.out‚25479.14697265625 disk.sda2.in‚0 disk.rbd1.out‚94.66943359375 disk.rbd2.in‚0 network.recv‚ 1.184664e+06* disk./.usagePercent‚36.649410103343484 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚96.97887897491455( proc.memory.availableMB‚ 494.98046875# proc.memory.rssMB‚ 2808.60546875 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd2.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.142367362976074 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 232.3046875 disk.sda.out‚ 143.35546875 disk.sda3.out‚ 143.35546875a¡²h2ƒ- _wandbruntime‚4424  _runtime‚4424–óÈ„Â trainer/global_step‚5199 $ _timestamp‚1.784277232217663e+09 _runtime‚4424.343298038 _step‚424 ' lr-AdamW/pg1‚0.00040613091000149646 ' lr-AdamW/pg2‚0.00040613091000149646ªÊíÍ…Ç $ _timestamp‚1.784277232217663e+09 _runtime‚4424.343298038 _step‚424 ' lr-AdamW/pg1‚0.00040613091000149646 ' lr-AdamW/pg2‚0.00040613091000149646 trainer/global_step‚5199¨ý8·2†- _wandbruntime‚4424  _runtime‚4424^’;ÿâ'‡ÜO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚794330057868800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚788843994873856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5544970835456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784794736119808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5477682531707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179689439256064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784794719080448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48113679 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚788843977834496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794329699490304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41138252 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20281860 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41138252 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚794330057868800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48113679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5544875966976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48103702 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚794329699490304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41138254 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5486062994944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48103702 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20281601 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚425 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9110118 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174144449808384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7842772322187476e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5486062994944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11171483 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5485721655808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174144468420608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5485721655808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚790272418651515 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚4424.344525074 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5477341981817 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚788843977834496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9110121 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚14381ÁEç'ˆáO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚794330057868800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48113679 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5544875966976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48103702 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚794329699490304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41138254 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5486062994944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48103702 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20281601 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚425 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9110118 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174144449808384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7842772322187476e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5486062994944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11171483 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5485721655808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174144468420608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5485721655808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚790272418651515 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚4424.344525074 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5477341981817 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚788843977834496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9110121 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚794330057868800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚788843994873856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5544970835456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784794736119808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5477682531707 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179689439256064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784794719080448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48113679 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚788843977834496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794329699490304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41138252 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20281860 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41138252 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576©”” 2‰- _wandbruntime‚4424  _runtime‚4424‹Üo5PŠåM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48122745 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268631 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.#J ›peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179723885401088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚794478311532544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48122745 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48112754 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9111863 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _step‚426 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41145884 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788991218864128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚788991235903488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174177897179136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 $ _timestamp‚1.784277232831703e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258638 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20285363 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5545988221952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5545895457792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11173500 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5478693585541 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41145886 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179723774024704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5487075629056 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9111866 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790419940377221 trainer/global_step‚5199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5487075629056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788991235903488 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚794477953146880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784941246791680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20285636 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48112754 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48122745 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268631 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179723885401088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚794478311532544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48122745 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48112754 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9111863 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _step‚426 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41145884 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788991218864128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚788991235903488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174177897179136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 $ _timestamp‚1.784277232831703e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258638 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20285363 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5545988221952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5545895457792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11173500 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5478693585541 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41145886 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179723774024704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5487075629056 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9111866 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790419940377221 trainer/global_step‚5199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5487075629056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788991235903488 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚794477953146880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784941246791680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20285636 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48112754 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23039604723453522 *jet/accuracy_2_of_4‚0.8096235394477844 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6255868077278137 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 'particle/accuracy_4_of_4‚ 0.94140625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6557671427726746 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18859247863292694 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13448816537857056 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 .particle/accuracy_3_of_4‚0.994953453540802 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023181866854429245 _runtime‚4442.007651628 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4937829375267029 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8610060811042786 +particle/specificity‚0.9093312621116638 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960885047912598 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.980837881565094 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5080746412277222 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927048683166504 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23174017667770386 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902437329292297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8564814329147339 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22598157823085785 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8023533821105957 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7872260808944702 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6402198076248169 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9719158411026001 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5877525210380554 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6302083134651184 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23348695039749146 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04035678505897522 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29346126317977905 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9915499091148376 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6425254344940186 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7063280344009399 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8703703284263611 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6832205653190613 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4915531277656555 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7705994844436646 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6701411604881287 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39146658778190613 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9151765704154968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7408195734024048 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7475363612174988 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47897493839263916 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6504064202308655 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8454201817512512 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666567325592 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9151512980461121 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6557671427726746 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6361784934997559 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6971051692962646 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7761386632919312 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8610060811042786 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7027021646499634 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9303771257400513 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.98896723985672 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6929484605789185 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21062591671943665 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7059577107429504 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.747287392616272 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23348695039749146 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7498721480369568 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14834579825401306 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32898157835006714 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8682528138160706 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7790627479553223 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7495101094245911 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9859579205513 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30763453245162964 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.980503261089325 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 particle/accuracy_3_of_3‚1 *particle/sensitivity‚0.611720085144043 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24074071645736694 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023181866854429245 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5598869323730469 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5674619674682617 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7230202555656433 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047113001346588135 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6704525947570801 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15001535415649414 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.749248206615448 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 *jet/accuracy_1_of_2‚0.7992290258407593 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2466181516647339 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02490670420229435 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18629378080368042 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7753213047981262 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7765684723854065 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6399171352386475 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24469752609729767 epoch‚12 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2003222405910492 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20203551650047302 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7671579718589783 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt0SRqqt*FBt/even[£ÌŽt_proportion‚0.0208565853536129 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029402760788798332 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03290584683418274 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8981198668479919 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5333333611488342 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7477315068244934 _step‚428 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944817423820496 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.746515154838562 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8552654981613159 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9844304919242859 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9899506568908691 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8016816973686218 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2248312383890152 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16417352855205536 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6295652389526367 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24510939419269562 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.031653132289648056 (loss/val_total_loss‚8.47570761425669 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 +jet/accuracy_2_of_2‚0.15959285199642181 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4637264013290405 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22989043593406677 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9784283638000488 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17825210094451904 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_1‚0.334643691778183 +jet/accuracy_3_of_4‚0.08837614953517914 6validation_average_jet_accuracy‚0.3781275749206543 $ _timestamp‚1.784277249881299e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22183683514595032 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960885047912598 +jet/accuracy_3_of_3‚0.06969916075468063 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957660436630249 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24074071645736694 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 trainer/global_step‚5199 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_4‚0.9717317223548889 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944981336593628 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8564814329147339 *jet/accuracy_2_of_3‚0.5756281614303589 particle/accuracy_1_of_1‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24398821592330933 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6552780866622925 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7415902018547058 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971743822097778 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6922959089279175 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6648343205451965 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05168292671442032 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.699253261089325 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0297901201993227 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.753242015838623 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6626570224761963 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4763106405735016 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03233029320836067 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.016844874247908592 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919431805610657 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21279923617839813uf÷ÄÁ=•»{ A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6295652389526367 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24510939419269562 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.031653132289648056 (loss/val_total_loss‚8.47570761425669 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 +jet/accuracy_2_of_2‚0.15959285199642181 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4637264013290405 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22989043593406677 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9784283638000488 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17825210094451904 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_1‚0.334643691778183 +jet/accuracy_3_of_4‚0.08837614953517914 6validation_average_jet_accuracy‚0.3781275749206543 $ _timestamp‚1.784277249881299e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22183683514595032 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960885047912598 +jet/accuracy_3_of_3‚0.06969916075468063 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957660436630249 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24074071645736694 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 trainer/global_step‚5199 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_4‚0.9717317223548889 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944981336593628 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8564814329147339 *jet/accuracy_2_of_3‚0.5756281614303589 particle/accuracy_1_of_1‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24398821592330933 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6552780866622925 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7415902018547058 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971743822097778 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6922959089279175 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6648343205451965 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05168292671442032 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.699253261089325 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0297901201993227 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.753242015838623 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6626570224761963 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4763106405735016 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03233029320836067 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.016844874247908592 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9919431805610657 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21279923617839813 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9929927587509155 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21401968598365784 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2929520308971405 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7040242552757263 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38856270909309387 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2029174417257309 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 particle/accuracy_2_of_2‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23039604723453522 *jet/accuracy_2_of_4‚0.8096235394477844 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6255868077278137 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 'particle/accuracy_4_of_4‚ 0.94140625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6557671427726746 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18859247863292694 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13448816537857056 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 .particle/accuracy_3_of_4‚0.994953453540802 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023181866854429245 _runtime‚4442.007651628 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4937829375267029 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8610060811042786 +particle/specificity‚0.9093312621116638 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960885047912598 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.980837881565094 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5080746412277222 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927048683166504 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23174017667770386 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902437329292297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8564814329147339 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22598157823085785 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8023533821105957 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7872260808944702 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6402198076248169 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9719158411026001 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5877525210380554 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6302083134651184 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23348695039749146 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04035678505897522 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29346126317977905 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9915499091148376 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6425254344940186 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7063280344009399 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8703703284263611 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6832205653190613 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4915531277656555 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7705994844436646 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6701411604881287 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39146658778190613 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9151765704154968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7408195734024048 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7475363612174988 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47897493839263916 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6504064202308655 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8454201817512512 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666567325592 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9151512980461121 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6557671427726746 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6361784934997559 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6971051692962646 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7761386632919312 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8610060811042786 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7027021646499634 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9303771257400513 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.98896723985672 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6929484605789185 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21062591671943665 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7059577107429504 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.747287392616272 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23348695039749146 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7498721480369568 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14834579825401306 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32898157835006714 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8682528138160706 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7790627479553223 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7495101094245911 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9859579205513 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30763453245162964 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.980503261089325 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 particle/accuracy_3_of_3‚1 *particle/sensitivity‚0.611720085144043 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24074071645736694 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023181866854429245 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5598869323730469 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5674619674682617 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7230202555656433 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047113001346588135 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6704525947570801 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15001535415649414 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.749248206615448 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 *jet/accuracy_1_of_2‚0.7992290258407593 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2466181516647339 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02490670420229435 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18629378080368042 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7753213047981262 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7765684723854065 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6399171352386475 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24469752609729767 epoch‚12 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2003222405910492 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20203551650047302 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7671579718589783 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029402760788798332 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03290584683418274 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8981198668479919 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5333333611488342 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7477315068244934 _step‚428 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944817423820496 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.746515154838562 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8552654981613159 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9844304919242859 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9899506568908691 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8016816973686218 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2248312383890152 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16417352855205536¬ çèž–‚(:“ ŽÒçÒÈ™¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jˆ+—‚(:  ŽÒçÒÌ·–§# proc.memory.rssMB‚ 2808.60546875$ disk./.usageGB‚546.0992736816406 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚94.79443359375 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875" disk.dm-0.out‚25500.8173828125 disk.rbd2.in‚0& proc.memory.availableMB‚ 495.171875 disk.dm-1.out‚ 144.46484375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.in‚0$ memory_percent‚96.97771072387695 disk.nvme0n1.in‚ 233.84765625 disk.drbd2165.out‚0* proc.memory.percent‚17.142367362976074 cpu‚26.02683418174399 network.sent‚ 8.263634e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚25500.8173828125 disk.dm-1.in‚0 disk.dm-0.in‚ 233.84765625 disk.rbd0.out‚0 network.recv‚ 1.195607e+06) disk./.usagePercent‚36.64918046458527 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 144.46484375 disk.rbd2.out‚0% disk.nvme0n1.out‚25500.8173828125" disk.nvme0c0n1.in‚ 233.84765625 disk.sda.out‚ 144.46484375¸\ñY‘˜‚(:† ÒçÒ˜Ñì¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚251.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>šÚŽ!™‚(:– ÒçÒûû§ disk.sda.out‚ 144.61328125$ memory_percent‚96.97740077972412* proc.memory.percent‚17.142367362976074 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 1.200157e+06 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚25510.9951171875 disk.sda3.out‚ 144.61328125# proc.memory.rssMB‚ 2808.60546875 cpu‚26.02574507069853 network.sent‚ 8.308203e+06 disk.dm-1.out‚ 144.61328125 disk.nvme0n1.in‚234.5625 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0) disk./.usagePercent‚36.64939781498184$ disk./.usageGB‚546.1025123596191" disk.dm-0.out‚25510.9951171875 disk.drbd2165.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 495.22265625 disk.rbd1.out‚94.79443359375 disk.sda1.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚234.5625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚25510.9951171875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚234.5625 disk.sda.in‚0û‹¾2š- _wandbruntime‚4481  _runtime‚4481Steɛà ' lr-AdamW/pg1‚0.00040480803553959426 ' lr-AdamW/pg2‚0.00040480803553959426 trainer/global_step‚5249 % _timestamp‚1.7842772892820852e+09 _runtime‚4481.407647243 _step‚429[¯ÈµÎœÈ _runtime‚4481.407647243 _step‚429 ' lr-AdamW/pg1‚0.00040480803553959426 ' lr-AdamW/pg2‚0.00040480803553959426 trainer/global_step‚5249 % _timestamp‚1.7842772892820852e+09­û:á¤2- _wandbruntime‚4481  _runtime‚4481ÍSÔ–îžÛO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798135595966464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚803690104708096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181446192447488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚4481.412358036 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48638453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798135613005824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803689746329600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20508024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41640518 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48628476 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚5249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚803690104708096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90278973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181446287053824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803689746329600 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11308392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175831799834624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41640520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚798135595966464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5554491702272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11308136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚799574951671332 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798135613005824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41640518 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5554491702272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5554150363136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9199891 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batcìVÈóh_start/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5614468606976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚799574594082082 % _timestamp‚1.7842772892830853e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5614392612864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚794028915888128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546018743844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5554150363136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚794028932927488Њˆæ'ŸàO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚4481.412358036 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48638453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798135613005824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803689746329600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20508024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41640518 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48628476 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚5249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚803690104708096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90278973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181446287053824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803689746329600 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11308392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175831799834624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41640520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚798135595966464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5554491702272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11308136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚799574951671332 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798135613005824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41640518 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5554491702272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5554150363136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9199891 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5614468606976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚799574594082082 % _timestamp‚1.7842772892830853e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5614392612864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚794028915888128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546018743844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5554150363136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚794028932927488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798135595966464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚803690104708096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181446192447488®Ï°¼ž2 - _wandbruntime‚4482  _runtime‚4482ûlòë&¡åM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚794175426560000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚431 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285678 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41648152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚798282836996096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175865247205376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚803838358371840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20511855 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5547029797678 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41648150 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5555504336384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚799722115807676 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48647519 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2H1²üð&¢êM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9201633 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48647519 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚803837999986176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5555504336384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚803838358371840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚5249 % _timestamp‚1.7842772898965352e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚798282836996096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175865247205376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚803838358371840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20511855 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5547029797678 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41648150 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5555504336384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚799722115807676 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48647519 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚794175426560000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚431 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285678 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41648152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EÅÝJ'§‚(:œ ¬ÒçÒÜð ¦& disk.nvme0n1.out‚25517.72802734375( disk.nvme0c0n1.out‚25517.72802734375 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚25517.72802734375* proc.memory.percent‚17.142367362976074 cpu‚26.024791445700462 disk.rbd0.out‚0( proc.memory.availableMB‚ 494.98046875 network.sent‚ 8.326316e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 235.390625 disk.sda2.in‚0# proc.memory.rssMB‚ 2808.60546875 disk.dm-1.out‚ 144.7734375 network.recv‚ 1.203433e+06 disk.nvme0n1.in‚ 235.390625 disk.rbd3.in‚0$ memory_percent‚96.97887897491455 proc.cpu.threads‚17# disk./.usageGB‚546.102466583252 disk.rbd2.in‚0 disk.sda.out‚ 144.7734375 disk.sda3.out‚ 144.7734375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚94.85693359375 disk.dm-0.in‚ 235.390625) disk./.usagePercent‚36.64939474289143 disk.rbd2.out‚0dù1T’¨‚(:‡ »ÒçÒÀµù¢ gpu.0.gpu‚10.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÀ“E&©‚(:› »ÒçÒ£Æþ¦ disk.rbd2.in‚0# proc.memory.rssMB‚ 2808.60546875 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0$ disk./.usageGB‚546.1006355285645 disk.rbd1.out‚94.85693359375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 145.19140625$ memory_percent‚96.98081016540527& disk.nvme0n1.out‚25523.36767578125 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.nvme0n1.in‚ 235.390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 235.390625 network.sent‚ 8.378413e+06 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚25523.36767578125' proc.memory.availableMB‚ 494.6640625 cpu‚26.02376587967013' disk./.usagePercent‚36.649271859275 disk.rbd0.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.142367362976074 disk.dm-1.in‚0 disk.dm-1.out‚ 145.19140625# disk.dm-0.out‚25523.36767578125 disk.rbd2.out‚0 disk.sda.out‚ 145.19140625 network.recv‚ 1.20794e+06 disk.dm-0.in‚ 235.390625ƒaŠª‘ª‚(:† ÊÒçÒѿΤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ýŠÈ+«‚(:  ÊÒçÒý°Œ§& disk.nvme0n1.out‚25531.10791015625 disk.dm-0.in‚ 236.10546875( proc.memory.availableMB‚ 494.66015625 cpu‚26.022754541812546( disk.nvme0c0n1.out‚25531.10791015625 disk.dm-1.out‚ 145.515625 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 8.381377e+06 disk.nvme0n1.in‚ 236.10546875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 network.recv‚ 1.210996e+06) disk./.usagePercent‚36.64928798774965$ disk./.usageGB‚546.1008758544922 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚ 0.01171875$ memory_percent‚96.98083400726318 disk.rbd2.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚94.91943359375# proc.memory.rssMB‚ 2808.60546875 disk.sda.in‚0 disk.sda.out‚ 145.515625 disk.sda3.out‚ 145.515625 disk.rbd3.out‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 236.10546875# disk.dm-0.out‚25531.10791015625* proc.memory.percent‚17.142367362976074X™;t2¬- _wandbruntime‚4521  _runtime‚4521jwP‡Ç­Á & lr-AdamW/pg1‚0.0004034743403120789 & lr-AdamW/pg2‚0.0004034743403120789 trainer/global_step‚5299 % _timestamp‚1.7842773290893214e+09 _runtime‚4521.214870887 _step‚433xYçÈÌ®Æ _runtime‚4521.214870887 _step‚433 & lr-AdamW/pg1‚0.0004034743403120789 & lr-AdamW/pg2‚0.0004034743403120789 trainer/global_step‚5299 % _timestamp‚1.7842773290893214e+09±,î¢>2¯- _wandbruntime‚4521  _runtime‚4521|¤ážâ'°ÜO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚806950744162590 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11412575 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801354449481728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801354466521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49092905 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805497647448064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5605187204096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49082928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42022118 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805497664487424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20699449 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811102493313024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183169266025472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20699716 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42022120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11412311 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177503237761024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5665921075200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚811102851691520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5605187204096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚811102851691520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚811102493313024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9287138 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5596635230752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5596294680862 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842773290902958e+09 _step‚434 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49082928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9287141 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚806951101751840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42022118 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5604845864960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5604845864960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚4521.219385891 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49092905 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177503256373248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438Ì:gç'±áO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚806950744162590 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11412575 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801354449481728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801354466521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49092905 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805497647448064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5605187204096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49082928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42022118 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805497664487424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20699449 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811102493313024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183169266025472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20699716 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42022120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11412311 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177503237761024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5665921075200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚811102851691520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5605187204096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚811102851691520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚811102493313024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9287138 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5596635230752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5596294680862 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842773290902958e+09 _step‚434 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49082928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9287141 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚806951101751840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42022118 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5604845864960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5604845864960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚4521.219385891 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49092905 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177503256373248²ÓÆQØ2²- _wandbruntime‚4521  _runtime‚4521—k‡²³æM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5597646284586 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42029752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚805644888477696 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚801500977192960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091980 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5606199838208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5667033076224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11414381 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177536666519552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49101971 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807098623477546 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚435 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20703532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183203718208000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91121730 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚4521.830944215 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842773297051005e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚811251105355264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.largk8~BÜe_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚805644905517056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9288886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11414646 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42029750 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20703264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5666942409216$¨&œñ&´ëM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91121730 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5605858491904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029750 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842773297051005e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚811251105355264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚805644905517056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9288886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11414646 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42029750 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20703264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5666942409216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5597646284586 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42029752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚805644888477696 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚801500977192960 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091980 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5606199838208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5667033076224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11414381 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177536666519552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49101971 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚807098623477546 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚435 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20703532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183203718208000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91121730 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚4521.830944215 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11516600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5655541366784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _step‚438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5655882705920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5717268298240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚812859698929664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9374391 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49537380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818515240296448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49537380 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚814327251832348 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚812859715969024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42403720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20890988 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11516855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179174694299648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941098 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚814326894243098 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808680000114688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49547357 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438wð+JÃáO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42403718 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42403720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _step‚438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5655882705920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5717268298240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚812859698929664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9374391 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49537380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818515240296448 G?DeviceStatsMonitor.on_train_batch_start/segment.smaô ll_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49537380 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚814327251832348 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚812859715969024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42403720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20890988 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11516855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179174694299648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941098 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚814326894243098 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808680000114688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49547357 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184891943985664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚5349 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11516600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5655541366784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42411352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42411350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9376136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9376133 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184926507411968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184926412812800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5349 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5656553993728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5718308366848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42411352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20894724 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818663852338688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49546432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49556423 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20894973 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808826493747200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656895340032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5648262771494 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚814474415968692 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49556423 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚813006956998656 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚818663852338688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5718384353792‹ä Oð&ÆêM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91957782 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179208123058176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20894724 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818663852338688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49546432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49556423 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20894973 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808826493747200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656895340032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5648262771494 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚814474415968692 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49556423 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚813006956998656 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚818663852338688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5718384353792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5656553993728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42411350 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42411352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42411350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9376136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9376133 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184926507411968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184926412812800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5349 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5656553993728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5718308366848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42411352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚820221750411264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180846132226048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5697868204568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11620623 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821703401912856 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5706236868608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚816005533708288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5706236868608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42785320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11620898 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821703044323606 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825927987279872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842774088841667e+09œ™áâç'ÓáO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚820221750411264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180846132226048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5697868204568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11620623 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821703401912856 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5706236868608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚816005533708288 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5706236868608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42785320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11620898 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821703044323606 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825927987279872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842774088841667e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5706578207744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21082539 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚820221767450624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚820221750411264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9461638 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825928345658368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508ºQÍE2Ô- _wandbruntime‚4601  _runtime‚4601çÐd,ë&ÕåM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816152027340800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821850923638562 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50000884 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186649109955584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 $ _timestamp‚1.784277409496752e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180879560984576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚820369008480256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚5399 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92793834 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5698879258402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚826076240936448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9463386 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚826076599322112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5698538708400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4601.622604907 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821850566049200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816152044380160 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826076599322112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21086065 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42792952 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826076599322112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21086065 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42792952 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816152027340800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821850923638562 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50000884 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186649109955584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 $ _timestamp‚1.784277409496752e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180879560984576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚820369008480256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚5399 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92793834 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5698879258402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚826076240936448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9463386 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚826076599322112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5698538708400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4601.622604907 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821850566049200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816152044380160 EïºÙ“Ü‚(:ˆ ³ÓçÒÏÖä¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÒÝã'Ý‚(:œ ³ÓçÒÇÇ‹§* disk./.usagePercent‚36.649562939841424 disk.nvme0c0n1.in‚ 240.734375 disk.sda3.in‚0 disk.dm-1.out‚ 147.2265625 disk.rbd2.out‚0 disk.rbd3.in‚0 cpu‚26.01586589567884 network.sent‚ 8.593079e+06 network.recv‚ 1.237871e+06& disk.nvme0n1.out‚25591.73095703125 disk.sda1.in‚0 disk.rbd1.out‚95.10693359375 disk.rbd3.out‚0# proc.memory.rssMB‚ 2808.60546875 disk.sda.in‚0$ memory_percent‚96.97794914245605* proc.memory.percent‚17.142367362976074 disk.sda1.out‚0( disk.nvme0c0n1.out‚25591.73095703125 disk.sda.out‚ 147.2265625$ disk./.usageGB‚546.1049728393555 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 147.2265625# disk.dm-0.out‚25591.73095703125' proc.memory.availableMB‚ 495.1328125 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 240.734375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 240.734375 disk.rbd1.in‚ 0.01171875IJ9P‘Þ‚(:† ÂÓçÒÔ²°¤ gpu.0.gpu‚6.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚286.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀR:*ß‚(:Ÿ ÂÓçҴ쿦 network.sent‚ 8.598276e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 240.734375$ memory_percent‚96.97959423065186* proc.memory.percent‚17.142367362976074 disk.rbd3.in‚0 disk.nvme0n1.in‚ 240.734375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 147.44921875 disk.sda2.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚546.1046676635742 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0# disk.dm-0.out‚25594.97607421875 disk.rbd2.in‚0( proc.memory.availableMB‚ 494.86328125( disk.nvme0c0n1.out‚25594.97607421875 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd2.out‚0# proc.memory.rssMB‚ 2808.60546875 network.recv‚ 1.24088e+06& disk.nvme0n1.out‚25594.97607421875 disk.dm-1.out‚ 147.44921875 cpu‚26.014860471349202 disk.nvme0c0n1.in‚ 240.734375 disk.sda3.out‚ 147.44921875) disk./.usagePercent‚36.64954245923869 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚95.16943359375§ˆ2à- _wandbruntime‚4641  _runtime‚4641ÇWÝÈá $ _timestamp‚1.784277448889487e+09 _runtime‚4641.015052592 _step‚445 ' lr-AdamW/pg1‚0.00039940919008530815 ' lr-AdamW/pg2‚0.00039940919008530815 trainer/global_step‚5449ØÂq«ÍâÇ $ _timestamp‚1.784277448889487e+09 _runtime‚4641.015052592 _step‚445 ' lr-AdamW/pg1‚0.00039940919008530815 ' lr-AdamW/pg2‚0.00039940919008530815 trainer/global_step‚5449½Óÿƒ[2ã- _wandbruntime‚4641  _runtime‚4641mºS9à'äÚO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5748484691476 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11724273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 trainer/global_step‚5449 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833340734263296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182517551540224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9548888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚833341092641792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4641.016395427 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43166920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50446284 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182517570152448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11724529 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5819818947584 _step‚446 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748144141586 $ _timestamp‚1.784277448890492e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5756932370432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823331050262528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5757273709568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43166920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188337290299392 bál^åßO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827583818932224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5756932370432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823331050262528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5757273709568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43166920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚188337290299392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5748484691476 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11724273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 trainer/global_step‚5449 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833340734263296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182517551540224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9548888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚833341092641792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4641.016395427 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43166920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50446284 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182517570152448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11724529 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5819818947584 _step‚446 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748144141586 $ _timestamp‚1.784277448890492e+09 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21277016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11726383 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚829226716129708 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188371752843776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5820850629120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50465327 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚823477560934400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43174550 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93629886 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827731042922496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21277279 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50455336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43174550 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50455336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 _runtime‚4641.627449965 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639879 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚447 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.784277449501485e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43174552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚827731059961856 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9550633 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚5449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182550998910976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11726643 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5749495745310 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚823477577973760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757944997376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚833488987919872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚833489346305536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188371849540096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50465327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50455336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 _runtime‚4641.627449965 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639879 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚447 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.784277449501485e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43174552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚827731059961856 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9550633 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚5449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182550998910976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11726643 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5749495745310 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚823477577973760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757944997376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚833488987919872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚833489346305536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188371849540096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50465327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21277016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11726383 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚829226716129708 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188371752843776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5820850629120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50465327 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚823477560934400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43174550 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169¿ªÊ¥?2é- _wandbruntime‚4641  _runtime‚4641Ü«Ø êƒ _runtime‚4641.629450021 _step‚448 0loss/FRt2/assignment_loss‚0.9838382945125044 2loss/SRqqt2/assignment_loss‚0.5247457680201701 1loss/SRqqt1/detection_loss‚1.1507948508594354 /loss/FBt1/detection_loss‚0.5590975956284263 epoch‚13 2loss/SRqqt1/assignment_loss‚0.5842356688161361 2loss/FBt1/assignment_loss‚0.036953892100509374 /loss/FRt1/detection_loss‚0.5359800019417161 1loss/SRqqt2/detection_loss‚0.6569613134256228 /loss/FBt2/detection_loss‚1.7568549197167158 %loss/total_loss‚9.922436926523007 trainer/global_step‚5449 % _timestamp‚1.7842774495041304e+09 0loss/FRt1/assignment_loss‚1.7692533631324767 1loss/FBt2/assignment_loss‚0.05307348518926925 /loss/FRt2/detection_loss‚1.3106477731800241OŠ0Uëˆ 2loss/SRqqt1/assignment_loss‚0.5842356688161361 2loss/FBt1/assignment_loss‚0.036953892100509374 /loss/FRt1/detection_loss‚0.5359800019417161 1loss/SRqqt2/detection_loss‚0.6569613134256228 /loss/FBt2/detection_loss‚1.7568549197167158 %loss/total_loss‚9.922436926523007 trainer/global_step‚5449 % _timestamp‚1.7842774495041304e+09 0loss/FRt1/assignment_loss‚1.7692533631324767 1loss/FBt2/assignment_loss‚0.05307348518926925 /loss/FRt2/detection_loss‚1.3106477731800241 _runtime‚4641.629450021 _step‚448 0loss/FRt2/assignment_loss‚0.9838382945125044 2loss/SRqqt2/assignment_loss‚0.5247457680201701 1loss/SRqqt1/detection_loss‚1.1507948508594354 /loss/FBt1/detection_loss‚0.5590975956284263 epoch‚13ÀÂûðБì‚(:† ÑÓçÒÌ¿§¥ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¼ í‚(:• ÑÓçÒÏÐק cpu‚26.013947465566346 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 147.6640625& disk.nvme0n1.out‚25604.98974609375# proc.memory.rssMB‚ 2808.60546875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚241.5625 disk.dm-0.in‚241.5625# disk.dm-0.out‚25604.98974609375 disk.sda1.out‚0# disk./.usageGB‚546.104808807373 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 147.6640625 disk.rbd1.in‚ 0.01171875 disk.rbd2.out‚0 network.sent‚ 8.60124e+06) disk./.usagePercent‚36.64955193151746( disk.nvme0c0n1.out‚25604.98974609375 disk.sda.in‚0$ memory_percent‚96.97797298431396 network.recv‚ 1.244338e+06 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 147.6640625 disk.rbd3.out‚0* proc.memory.percent‚17.142367362976074 disk.rbd1.out‚95.16943359375 disk.rbd3.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 495.12890625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚241.5625¨šáˆžî‚(:“ àÓçÒäÏ‹¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚288.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.668600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E-ï‚(:¢ àÓçÒØ瘧 disk.sda2.out‚0& disk.nvme0n1.out‚25612.15771484375 disk.drbd2165.out‚0* proc.memory.percent‚17.142367362976074 disk.rbd1.out‚95.23193359375 disk.nvme0n1.in‚ 242.27734375 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 147.7734375 disk.dm-1.out‚ 147.7734375# disk.dm-0.out‚25612.15771484375 disk.drbd2165.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2808.60546875$ disk./.usageGB‚546.0983047485352" disk.nvme0c0n1.in‚ 242.27734375 disk.sda3.in‚0 disk.rbd2.out‚0 cpu‚26.013059466584167 disk.sda.in‚0 disk.dm-0.in‚ 242.27734375 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0$ memory_percent‚96.97680473327637 network.sent‚ 8.664041e+06) disk./.usagePercent‚36.64911543867157( disk.nvme0c0n1.out‚25612.15771484375 disk.sda.out‚ 147.7734375 disk.rbd1.in‚ 0.01171875' proc.memory.availableMB‚ 495.3203125 network.recv‚ 1.248255e+06 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0Š9[’ð‚(:’ ïÓçҊႤ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öØ÷ ñ‚(: ïÓçÒÚ‘€¦ disk.sda2.out‚0 disk.dm-0.in‚ 243.10546875 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 243.10546875 disk.sda.out‚ 148.0859375# memory_percent‚96.9782829284668& proc.memory.availableMB‚ 495.078125 disk.dm-1.out‚ 148.0859375 disk.rbd3.out‚0 disk.dm-0.out‚ 25620.234375! disk.nvme0n1.out‚ 25620.234375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.out‚ 25620.234375 disk.sda3.in‚0 disk.rbd2.out‚0 network.recv‚ 1.251833e+06 disk.rbd1.out‚95.23193359375 disk.rbd0.out‚0 disk.sda3.out‚ 148.0859375* proc.memory.percent‚17.142367362976074 network.sent‚ 8.670533e+06 disk.rbd1.in‚ 0.01171875 proc.cpu.threads‚17$ disk./.usageGB‚546.0981636047363 disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 243.10546875 disk.sda1.in‚0# proc.memory.rssMB‚ 2808.60546875 cpu‚26.012132419811632( disk./.usagePercent‚36.6491059663928 disk.rbd2.in‚0 disk.sda.in‚0|NÚ×2ò- _wandbruntime‚4681  _runtime‚4681ÌÂèQÉóà % _timestamp‚1.7842774889700313e+09 _runtime‚4681.097452638 _step‚449 ' lr-AdamW/pg1‚0.00039803307527004503 ' lr-AdamW/pg2‚0.00039803307527004503 trainer/global_step‚5499o2PVÎôÈ ' lr-AdamW/pg1‚0.00039803307527004503 ' lr-AdamW/pg2‚0.00039803307527004503 trainer/global_step‚5499 % _timestamp‚1.7842774889700313e+09 _runtime‚4681.097452638 _step‚449Á‡°Í2õ- _wandbruntime‚4681  _runtime‚4681€¢¾tx öÜO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840753481246720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11828639 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5798760628494 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43548520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840753839625216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190060097224192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚836455344484622 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43548518 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50910713 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_traiç: ¬jn_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21464777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _step‚450 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50910713 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚830656600895488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449254 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50910713 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚830656600895488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449254 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840753481246720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11828639 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5798760628494 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43548520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840753839625216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190060097224192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚836455344484622 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43548518 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50910713 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154Â{ü³2ø- _wandbruntime‚4681  _runtime‚4681Ôý¶ë&ùåM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21468591 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184222436837376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43556152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9637883 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚840902093288960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9637886 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835093094404096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94465938 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11830962 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190094658545664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚836603223799578 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475931 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50919779 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43556152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50909788 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840901734903296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚836602866210216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚840902093288960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190094559752192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5800112232218 % _timestamp‚1.7842774895819404e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚5499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚451 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50919779 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43556150 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184222418225152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835093094404096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5808640499200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0m]½\oúêM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840901734903296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚836602866210216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚840902093288960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190094559752192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5800112232218 % _timestamp‚1.7842774895819404e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚5499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚451 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50919779 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43556150 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184222418225152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚835093094404096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5808640499200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21468591 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184222436837376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43556152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9637883 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚840902093288960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9637886 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835093094404096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94465938 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11830962 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190094658545664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚836603223799578 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475931 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50919779 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43556152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50909788Ãq eù2û- _wandbruntime‚4681  _runtime‚4681Vcø¤ üƒ epoch‚13 0loss/FRt2/assignment_loss‚0.9147102909592482 2loss/SRqqt1/assignment_loss‚0.5537775664480065 1loss/FBt2/assignment_loss‚0.20561878390506827 /loss/FBt1/detection_loss‚0.5613598647320779 _runtime‚4681.710411666 2loss/FBt1/assignment_loss‚0.026688014642554315 1loss/SRqqt1/detection_loss‚0.9362400909329444 % _timestamp‚1.7842774895850766e+09 0loss/FRt1/assignment_loss‚1.7824258515111457 /loss/FRt1/detection_loss‚0.5216482829128265 /loss/FRt2/detection_loss‚1.3789579367448683 /loss/FBt2/detection_loss‚0.8524740941356868 trainer/global_step‚5499 2loss/SRqqt2/assignment_loss‚0.5540410813818614 1loss/SRqqt2/detection_loss‚0.6996998492252746 _step‚452 %loss/total_loss‚8.987641707531562¹ µ»ýˆ 2loss/SRqqt1/assignment_loss‚0.5537775664480065 1loss/FBt2/assignment_loss‚0.20561878390506827 /loss/FBt1/detection_loss‚0.5613598647320779 _runtime‚4681.710411666 2loss/FBt1/assignment_loss‚0.026688014642554315 1loss/SRqqt1/detection_loss‚0.9362400909329444 % _timestamp‚1.7842774895850766e+09 0loss/FRt1/assignment_loss‚1.7824258515111457 /loss/FRt1/detection_loss‚0.5216482829128265 /loss/FRt2/detection_loss‚1.3789579367448683 /loss/FBt2/detection_loss‚0.8524740941356868 trainer/global_step‚5499 2loss/SRqqt2/assignment_loss‚0.5540410813818614 1loss/SRqqt2/detection_loss‚0.6996998492252746 _step‚452 %loss/total_loss‚8.987641707531562 epoch‚13 0loss/FRt2/assignment_loss‚0.9147102909592482Ä ]“þ‚(:ˆ þÓçÒ¯“ë¤ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.984' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fìŽ+ÿ‚(:  þÓçÒÚ‡ö¦ cpu‚26.01118689346986 disk.dm-1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚546.0985870361328 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 148.3046875 disk.rbd1.in‚ 0.01171875 network.sent‚ 8.686694e+06# disk.dm-0.out‚25627.09521484375 disk.sda3.out‚ 148.3046875# proc.memory.rssMB‚ 2808.60546875 disk.dm-0.in‚ 243.8203125& disk.nvme0n1.out‚25627.09521484375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚ 1.255397e+06 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚36.649134383229104 disk.rbd0.out‚0 disk.sda.out‚ 148.3046875 disk.rbd1.out‚95.29443359375( proc.memory.availableMB‚ 495.26171875* proc.memory.percent‚17.142367362976074 disk.nvme0n1.in‚ 243.8203125( disk.nvme0c0n1.out‚25627.09521484375 disk.sda.in‚0$ memory_percent‚96.97716236114502 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 243.8203125ˆ@¦ž€‚(:“ ÔçÒØž’¤ gpu.0.gpu‚63.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃRéú#‚(:˜ Ôç҇Ѽ¦ network.sent‚ 8.738808e+06 disk.nvme0n1.in‚ 243.8203125 disk.sda.out‚ 148.48828125 disk.drbd2165.out‚0* proc.memory.percent‚17.142367362976074! disk.nvme0c0n1.in‚ 243.8203125 disk.sda3.out‚ 148.48828125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2808.60546875) disk./.usagePercent‚36.64912977509349 disk.sda1.in‚0" proc.memory.availableMB‚495.25 cpu‚26.01031570192287 proc.cpu.threads‚17% disk.nvme0n1.out‚25633.5654296875' disk.nvme0c0n1.out‚25633.5654296875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚95.29443359375 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 243.8203125 network.recv‚ 1.259735e+06" disk.dm-0.out‚25633.5654296875$ memory_percent‚96.97723388671875# disk./.usageGB‚546.098518371582 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 148.48828125 disk.rbd1.in‚ 0.01171875?4A%2‚- _wandbruntime‚4720  _runtime‚4720»r¸ ǃÁ trainer/global_step‚5549 % _timestamp‚1.7842775287384827e+09 _runtime‚4720.864051192 _step‚453 & lr-AdamW/pg1‚0.0003966465748322067 & lr-AdamW/pg2‚0.0003966465748322067[“DÂÌ„Æ _runtime‚4720.864051192 _step‚453 & lr-AdamW/pg1‚0.0003966465748322067 & lr-AdamW/pg2‚0.0003966465748322067 trainer/global_step‚5549 % _timestamp‚1.7842775287384827e+09ÅÄõ+?2…- _wandbruntime‚4720  _runtime‚4720VhlEâ'†ÜO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚848166586608640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5858664713216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43930118 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843831852154380 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43930118 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842307921895424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚454 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842307921895424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191782975613952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9723391 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5922529608704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849717665292 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5858323374080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51355188 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11933065 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5549 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚848166586608640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21656192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51365165 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295285 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9723388 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285306 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5849377115402 _runtime‚4720.868379722 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842775287394977e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5858323374080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185860427393024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5858664713216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843831494565130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51355188 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43930120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842307904856064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11932804 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848166228230144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21656456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5922445225984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976„V²ç'‡áO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43930118 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843831852154380 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43930118 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842307921895424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚454 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842307921895424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191782975613952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9723391 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5922529608704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849717665292 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5858323374080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51355188 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11933065 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5549 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚848166586608640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21656192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51365165 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295285 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9723388 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285306 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5849377115402 _runtime‚4720.868379722 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842775287394977e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5858323374080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185860427393024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5858664713216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843831494565130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51355188 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43930120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842307904856064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11932804 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848166228230144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21656456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5922445225984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚848166586608640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5858664713216Æ– S2ˆ- _wandbruntime‚4721  _runtime‚4721Œßc2U ‰æM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43937750 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51364240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191817319135744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚848314840272384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚848314481886720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚843979373880086 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43937752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842455145885696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191817426317824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51364240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842455162925056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5850388169124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5859677347328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11935178 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_™ ÙÖ—pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301990 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5859336001024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51374231 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185893874763776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43937750 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚4721.479545987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21660055 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚838128645160960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚842455145885696 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7842775293536952e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95311983 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9725136 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5859336001024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚842455162925056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843979016290724 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43937752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51374231 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43937752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842455145885696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191817426317824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51364240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842455162925056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5850388169124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5859677347328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11935178 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301990 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5859336001024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51374231 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185893874763776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43937750 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚4721.479545987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21660055 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚838128645160960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚842455145885696 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7842775293536952e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95311983 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9725136 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5859336001024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚842455162925056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843979016290724 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43937752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51374231 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96131337 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842775687310214e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5909018875904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚849669973377024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚849669956337664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12036513 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚4760.856779483 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44311720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44311718 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚845307651043328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855578975213568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51809640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855579333592064 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;Devicþ³GžJeStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51809640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44311718 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169ùžõç'™áO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51819617 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9810641 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚851207644645638 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5899993602310 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚849669956337664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855578975213568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187531883931648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96121358 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12036249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187531865319424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44311720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193505577775616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51809640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44311718 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96131337 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842775687310214e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5909018875904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚849669973377024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚849669956337664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12036513 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚4760.856779483 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44311720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44311718 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚845307651043328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855578975213568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51809640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855579333592064 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚845454161715200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44319350 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9812383 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚5599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44319350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9812386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12038335 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193540136717312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚849817214406656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5974739651584 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21850979 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187565312690176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193540033729536 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51828683 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855727587255808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51818692 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5910372849152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855727228870144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21850718 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21850718 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚845454161715200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44319350 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9812383 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚5599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44319350 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9812386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12038335 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193540136717312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138042 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚849817214406656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5974739651584 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21850979 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187565312690176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193540033729536 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51828683 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855727587255808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51818692 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5910372849152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855727228870144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269ËÈŽ†b2- _wandbruntime‚4761  _runtime‚4761ƒš9Nž 2loss/SRqqt1/assignment_loss‚0.5173114182701241 /loss/FRt2/detection_loss‚1.2798257953893606 1loss/SRqqt2/detection_loss‚0.6381320934364487 %loss/total_loss‚8.367228927984055 epoch‚13 _step‚460 1loss/SRqqt2/assignment_loss‚0.517520462539402 /loss/FBt1/detection_loss‚0.3981175776868842 /loss/FBt2/detection_loss‚0.7290706180568253 0loss/FRt1/assignment_loss‚1.7442202616932723 /loss/FRt2/assignment_loss‚0.989920001373714 2loss/FBt2/assignment_loss‚0.004118990197327678 trainer/global_step‚5599 % _timestamp‚1.7842775693464203e+09 _runtime‚4761.471747207 1loss/FBt1/assignment_loss‚0.03154370737451734 /loss/FRt1/detection_loss‚0.5344510820398336 1loss/SRqqt1/detection_loss‚0.9829969199263446€ ÐmOŸ† 1loss/FBt1/assignment_loss‚0.03154370737451734 /loss/FRt1/detection_loss‚0.5344510820398336 1loss/SRqqt1/detection_loss‚0.9829969199263446 2loss/SRqqt1/assignment_loss‚0.5173114182701241 /loss/FRt2/detection_loss‚1.2798257953893606 1loss/SRqqt2/detection_loss‚0.6381320934364487 %loss/total_loss‚8.367228927984055 epoch‚13 _step‚460 1loss/SRqqt2/assignment_loss‚0.517520462539402 /loss/FBt1/detection_loss‚0.3981175776868842 /loss/FBt2/detection_loss‚0.7290706180568253 0loss/FRt1/assignment_loss‚1.7442202616932723 /loss/FRt2/a>ÿÕ>½ssignment_loss‚0.989920001373714 2loss/FBt2/assignment_loss‚0.004118990197327678 trainer/global_step‚5599 % _timestamp‚1.7842775693464203e+09 _runtime‚4761.471747207Ì®fל ‚(:‘ ÉÔçÒب¥ gpu.0.gpu‚36.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôJ¢o+¡‚(:  ÉÔçÒà™Ç§ disk.rbd3.out‚0( proc.memory.availableMB‚ 494.98046875 disk.sda3.out‚ 149.515625 disk.dm-1.out‚ 149.515625 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚25665.044921875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 246.19140625! disk.dm-0.out‚25665.044921875$ memory_percent‚96.97887897491455 proc.cpu.threads‚17 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2808.60546875$ disk.nvme0n1.out‚25665.044921875) disk./.usagePercent‚36.64930360420924 disk.rbd0.in‚0 disk.rbd1.out‚95.41943359375 disk.nvme0n1.in‚ 246.19140625* proc.memory.percent‚17.142367362976074 network.sent‚ 8.828079e+06$ disk./.usageGB‚546.1011085510254" disk.nvme0c0n1.in‚ 246.19140625 disk.sda.in‚0 disk.sda.out‚ 149.515625 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0.140625 network.recv‚ 1.274144e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda1.out‚0 cpu‚26.01328983661254d§¬2¢- _wandbruntime‚4778  _runtime‚4778ͦÑ)”=£Ž{ 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2810550034046173 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7598374485969543 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.754607081413269 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_4‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5180743336677551 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28922659158706665 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8709853887557983 (particle/accuracy‚0.8468735218048096 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7933862805366516 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22995050251483917 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05419350415468216 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7984137535095215 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166668057441711 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15097485482692719 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4925249218940735 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7096664905548096 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04199220985174179 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8622347712516785 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6328744292259216 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.060561973601579666 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.404885470867157 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7838419675827026 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02426188625395298 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7564034461975098 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5941792130470276 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29973989725112915 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6933483481407166 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7220354080200195 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955105781555176 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22064082324504852 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7783671617507935 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6786788105964661 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3636443614959717 trainer/global_step‚5599 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23245342075824738 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3267309069633484 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973524212837219 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9968619346618652 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11784368753433228 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03076385334134102 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07332945615053177 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0259100254625082 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7967498302459717 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _step‚461 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2482295036315918 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9644074440002441 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5007561445236206 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04661251977086067 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7626994848251343 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2397339940071106 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24516351521015167 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32868334650993347 particle/accuracy_3_of_4‚1 +particle/specificity‚0.9107107520103455 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6632007360458374 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2916471064090729 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7865458130836487 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997683048248291 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9228391051292419 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40206649899482727 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9945387840270996 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8020220398902893 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5541666746139526 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916865348816 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 6"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7528116106987 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6724469065666199 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21277053654193878 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_2‚0.8044489622116089 particle/accuracy_3_of_5‚1 _runtime‚ 4778.56704925 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 epoch‚13 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6663634181022644 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.637810230255127 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6760644912719727 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5721102356910706 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.40011247992515564 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877297282218933 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893644452095032 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022759411484003067 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2775188088417053 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898260235786438 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8004138469696045 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30170100927352905 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6244314908981323 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9968750476837158 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 3 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990966796875 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026143454015254974 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_2_of_4‚0.8042323589324951 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7536961436271667 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23675042390823364 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3441949188709259 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8517159223556519 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7847694158554077 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7555567622184753 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.08864892274141312 +particle/sensitivity‚0.6134400963783264 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25072184205055237 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2261039912700653 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.301340788602829 particle/accuracy_3_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 )loss/val_total_loss‚8.358637069550113 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5046296119689941 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5698404908180237 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6786788105964661 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022520657628774643 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26148927211761475 particle/accuracy_1_of_3‚1 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.761902391910553 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7331600785255432 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7181217670440674 jet/accuracy_3_of_5‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22859756648540497 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962199330329895 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5331078767776489 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9968750476837158 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 $ _timestamp‚1.784277586440809e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7116891741752625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6824765801429749 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8517159223556519 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7196189761161804 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6862473487854004 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7228618860244751 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6340301632881165 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7583421468734741 *jet/accuracy_1_of_3‚0.9318200945854187 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13335224986076355 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23681537806987762 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7028704881668091 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950046539306641 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946612119674683 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_2‚0.1646452695131302 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6592774987220764 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2916471064090729 particle/accuracy_2_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3623588979244232 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4750048816204071 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7545866966247559 /particle/accuracy_4_of_4‚0.9387219548225403 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6324127912521362 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7140238881111145 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5036318898200989 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13493911921977997 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03147447854280472 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03076385334134102 *jet/accuracy_2_of_3‚0.5838314890861511~»6§Œ;¤“{ 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7838419675827026 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02426188625395298 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7564034461975098 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5941792130470276 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29973989725112915 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6933483481407166 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7220354080200195 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955105781555176 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22064082324504852 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7783671617507935 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6786788105964661 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3636443614959717 trainer/global_step‚5599 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23245342075824738 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3267309069633484 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973524212837219 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9968619346618652 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11784368753433228 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03076385334134102 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07332945615053177 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0259100254625082 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7967498302459717 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _step‚461 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2482295036315918 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_4‚0.9644074440002441 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5007561445236206 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04661251977086067 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7626994848251343 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2397339940071106 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24516351521015167 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32868334650993347 particle/accuracy_3_of_4‚1 +particle/specificity‚0.9107107520103455 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6632007360458374 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2916471064090729 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7865458130836487 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997683048248291 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9228391051292419 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40206649899482727 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9945387840270996 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8020220398902893 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5541666746139526 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916865348816 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 6"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7528116106987 particle/accuracy_2_of_4‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6724469065666199 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21277053654193878 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_2‚0.8044489622116089 particle/accuracy_3_of_5‚1 _runtime‚ 4778.56704925 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 epoch‚13 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6663634181022644 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.637810230255127 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6760644912719727 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5721102356910706 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.40011247992515564 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877297282218933 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893644452095032 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022759411484003067 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2775188088417053 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898260235786438 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8004138469696045 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30170100927352905 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6244314908981323 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9968750476837158 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 3 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990966796875 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026143454015254974 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_2_of_4‚0.8042323589324951 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7536961436271667 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23675042390823364 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3441949188709259 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8517159223556519 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7847694158554077 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7555567622184753 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.08864892274141312 +particle/sensitivity‚0.6134400963783264 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25072184205055237 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2261039912700653 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.301340788602829 particle/accuracy_3_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 )loss/val_total_loss‚8.358637069550113 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5046296119689941 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5698404908180237 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6786788105964661 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022520657628774643 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26148927211761475 particle/accuracy_1_of_3‚1 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.761902391910553 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7331600785255432 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7181217670440674 jet/accuracy_3_of_5‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22859756648540497 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962199330329895 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5331078767776489 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9968750476837158 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 $ _timestamp‚1.784277586440809e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7116891741752625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6824765801429749 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8517159223556519 /particle/accuracy_2_of_2‚0.9998962879180908 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7196189761161804 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6862473487854004 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7228618860244751 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6340301632881165 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7583421468734741 *jet/accuracy_1_of_3‚0.9318200945854187 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13335224986076355 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23681537806987762 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7028704881668091 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950046539306641 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946612119674683 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_2‚0.1646452695131302 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6592774987220764 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2916471064090729 particle/accuracy_2_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3623588979244232 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4750048816204071 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7545866966247559 /particle/accuracy_4_of_4‚0.9387219548225403 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6324127912521362 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7140238881111145 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5036318898200989 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13493911921977997 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03147447854280472 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03076385334134102 *jet/accuracy_2_of_3‚0.5838314890861511 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2810550034046173 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7598374485969543 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.754607081413269 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_4‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5180743336677551 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28922659158706665 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8709853887557983 (particle/accuracy‚0.8468735218048096 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7933862805366516 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22995050251483917 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05419350415468216 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7984137535095215 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166668057441711 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15097485482692719 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4925249218940735 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7096664905548096 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04199220985174179 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8622347712516785 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6328744292259216 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.060561973601579666 8"Purity/*FRt1SRqqt*FBt/eveý nt_purity‚0.404885470867157 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375ͪ-‰Üž¥‚(:“ ØÔçÒäϸ¥ gpu.0.gpu‚60.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àáú(:ã ØÔçÒ³Û᧠disk.rbd0.in‚0 disk.sda.out‚ 151.49609375 disk.sda3.in‚0 network.recv‚ 1.278687e+06$ disk./.usageGB‚546.1088790893555 disk.nvme0c0n1.in‚ 246.90625 disk.sda3.out‚ 151.49609375 disk.rbd1.out‚95.48193359375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 151.49609375 disk.rbd3.out‚0 disk.sda1.out‚0$ memory_percent‚95.46446800231934# proc.memory.rssMB‚ 2808.62890625 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 246.90625 cpu‚26.01883591302832 disk.dm-0.in‚ 246.90625 disk.dm-0.out‚ 25924.921875' proc.memory.availableMB‚ 743.1015625 network.sent‚ 8.890956e+06# disk.nvme0c0n1.out‚ 25924.921875 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0) disk./.usagePercent‚36.64982509155649! disk.nvme0n1.out‚ 25924.921875* proc.memory.percent‚17.142510414123535¦ì>!ž§‚(:“ çÔçÒ‹™Î¤ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìû÷í¨‚(:â çÔçÒù²û¦ disk.sda2.out‚0$ memory_percent‚96.96729183197021 disk.sda3.out‚ 152.96875 disk.rbd1.in‚ 0.01171875 network.sent‚ 8.935376e+06 disk.nvme0n1.in‚ 247.734375 disk.nvme0c0n1.in‚ 247.734375* proc.memory.percent‚17.142510414123535 disk.dm-0.in‚ 247.734375 disk.dm-0.out‚ 26180.3984375( proc.memory.availableMB‚ 496.87890625 disk.drbd2165.in‚0 proc.cpu.threads‚17" disk.nvme0n1.out‚ 26180.3984375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚95.48193359375 disk.rbd3.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 26180.3984375 disk.sda.out‚ 152.96875 disk.sda3.in‚0) disk./.usagePercent‚36.64954143520855 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2808.62890625$ disk./.usageGB‚546.1046524047852 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 152.96875 cpu‚26.017909252076418 network.recv‚ 1.282649e+06Òö•“©‚(:ˆ öÔçÒÃôݤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð𠪂(:ü öÔçÒù߉© disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 248.44921875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 155.09765625 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚95.54443359375 disk.rbd3.out‚0) disk./.usagePercent‚36.61746625124358# disk./.usageGB‚545.626708984375 disk.drbd2165.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚26192.09814453125# disk.dm-0.out‚26192.09814453125# proc.memory.rssMB‚ 2808.62890625* proc.memory.percent‚17.142510414123535 disk.nvme0n1.in‚ 248.44921875 disk.dm-0.in‚ 248.44921875( proc.memory.availableMB‚ 496.85546875 cpu‚26.016853030189026 network.sent‚ 8.940376e+06$ memory_percent‚96.96743488311768 network.recv‚ 1.286013e+06 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 155.09765625 disk.dm-1.out‚ 155.09765625 disk.rbd1.in‚ 0.01171875& disk.nvme0n1.out‚26192.09814453125 disk.drbd2165.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17„eç2«- _wandbruntime‚4819  _runtime‚4819cq›Vɬà ' lr-AdamW/pg2‚0.00039384277395946297 trainer/global_step‚5649 % _timestamp‚1.7842776277529042e+09 _runtime‚4819.878535378 _step‚462 ' lr-AdamW/pg1‚0.000393842773959462977ËöšÎ­È ' lr-AdamW/pg1‚0.00039384277395946297 ' lr-AdamW/pg2‚0.00039384277395946297 trainer/global_step‚5649 % _timestamp‚1.7842776277529042e+09 _runtime‚4819.878535378 _step‚462ΊÎ82®- _wandbruntime‚4819  _runtime‚4819 ! â'¯ÜO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12172033 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52344396 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _step‚463 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚860510177665460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚858961574469632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9900408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚860510535254710 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚864939380433920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5977447585792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195262443947520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚854541847851008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5968329814452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842776277540412e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158382 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22072444 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44813984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6043127704064 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44813986 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44813984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52344396 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189219233957888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚864939022055424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6043209989632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44813986 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52334419 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚854541864890368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚858961591508992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚858961574469632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195262343049728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5977447585792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22072173 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9900411 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚864939022055424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12171765 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44813984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6043127704064 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44813986 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44813984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52344396 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189219233957888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚864939022055424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6043209989632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44813986 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52334419 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚854541864890368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚858961591508992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚858961574469632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195262343049728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5977447585792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22072173 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9900411 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚864939022055424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12171765 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195262443947520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚854541847851008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5968329814452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888Ïð~¯Ê2±- _wandbruntime‚4820  _runtime‚4820ÏÄ«¿ã²åM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5978460212736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969681418176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5978801559040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22076274 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9902156 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6044150463488 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97165087 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚865087634097664 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚4820.489877187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44821618 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5969340868174 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52353462 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44821618 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52343471 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195296794567680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚865087275712000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859108815499264 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22076007 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9902153 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189252644104192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195296897555456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocaTñdx ted‚ 6044234839040 _step‚464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97165087 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚865087634097664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44821616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189252662716416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚854688358522880 $ _timestamp‚1.784277628364061e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12174118 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚859108832538624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚854688375562240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44821616 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚865087634097664 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚4820.489877187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44821618 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5969340868174 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52353462 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44821618 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52343471 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195296794567680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚865087275712000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859108815499264 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22076007 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9902153 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189252644104192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195296897555456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6044234839040 _step‚464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97165087 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚865087634097664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44821616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189252662716416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚854688358522880 $ _timestamp‚1.784277628364061e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12174118 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚859108832538624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚854688375562240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44821616 Fù1KǼÁ % _timestamp‚1.7842776676357198e+09 _runtime‚4859.761297066 _step‚466 & lr-AdamW/pg1‚0.0003924256536074483 & lr-AdamW/pg2‚0.0003924256536074483 trainer/global_step‚5699ûÔ²zÌ½Æ _runtime‚4859.761297066 _step‚466 & lr-AdamW/pg1‚0.0003924256536074483 & lr-AdamW/pg2‚0.0003924256536074483 trainer/global_step‚5699 % _timestamp‚1.7842776676357198e+09ÒÚ—äÖ2¾- _wandbruntime‚4859  _runtime‚4859F—®ôâ'¿ÜO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45195586 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚872352127417344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚866323625951232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚866323625951232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196985394599424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚467 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6028484426752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872351769038848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9987661 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190890671884288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867886327745968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45195584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9987658 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚866323642990592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872352127417344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196985291604480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12277368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45195586 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7842776676367414e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52788871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12277627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190890653272064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6028484426752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6094638332416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861867398483968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872351769038848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6028143087616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867886685335218 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52788871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6019286851250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6018946301360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52798848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚4859.762504225 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚866323642990592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22265288 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7842776676367414e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52788871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12277627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190890653272064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6028484426752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6094638332416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861867398483968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872351769038848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6028143087616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867886685335218 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52788871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6019286851250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6018946301360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52798848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚4859.762504225 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]@pÌ«šIDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚866323642990592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22265288 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45195584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9987658 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚866323642990592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Ó§œóžÁ‚(:“ £ÕçÒ®‹ö¤ gpu.0.gpu‚38.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(av‚(:÷ £ÕçÒµš©¥ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚26221.21630859375# disk.dm-0.out‚26221.21630859375$ disk./.usageGB‚545.6271095275879 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0# memory_percent‚96.9700813293457 proc.cpu.threads‚17 network.sent‚ 9.013465e+06 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 249.9921875 disk.sda.in‚0 disk.rbd1.out‚95.60693359375 disk.dm-0.in‚ 249.9921875 disk.sda3.out‚ 156.01171875 disk.rbd1.in‚ 0.01171875& proc.memory.availableMB‚ 496.421875 disk.sda1.out‚0# proc.memory.rssMB‚ 2808.62890625 cpu‚26.014141019583295) disk./.usagePercent‚36.61749313203468 disk.sda.out‚ 156.01171875 disk.dm-1.out‚ 156.01171875 disk.rbd3.in‚0* proc.memory.percent‚17.142510414123535 disk.nvme0n1.in‚ 249.9921875& disk.nvme0n1.out‚26221.21630859375 network.recv‚ 1.297449e+06 disk.drbd2165.in‚0Ùþ&2Ã- _wandbruntime‚4860  _runtime‚48607x)pì&ÄæM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚866470884020224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚468 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22268843 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚866470866980864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22269106 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6095647935488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9989403 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚862013892116480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52807914 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197019729966080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚866470884020224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45203218 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6029497060864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚872500381081088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45203216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6095740699648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862013909155840 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190924100642816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚868034207060924 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011132 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52797923 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842776682511325e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12279440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190924082030592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45203216 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842776682511325e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12279440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190924082030592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45203216 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22268843 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚866470866980864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22269106 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6095647935488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9989403 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚862013892116480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52807914 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197019729966080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚866470884020224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45203218 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6029497060864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚872500381081088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45203216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6095740699648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862013909155840 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190924100642816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚868034207060924 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011132 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52797923 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.oéT&n_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6078838589440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12382565 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45577186 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22457473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10074911 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53253300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98820507 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚873685694472192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6146123580416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22457733 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198708315676672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚471 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192562091198464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198708214778880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879764516022272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879764516022272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6069562788268 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45577186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875262835415726 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45577184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚869192915038208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12382822 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879764874400768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192562109810688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12382822 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879764874400768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192562109810688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6078838589440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12382565 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45577186 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22457473 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10074911 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53253300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98820507 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚873685694472192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6146123580416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22457733 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198708315676672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚471 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192562091198464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198708214778880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879764516022272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879764516022272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6069562788268 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45577186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875262835415726 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976׎*2Ó- _wandbruntime‚4900  _runtime‚4900G8tî&ÔèM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6080192562688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45584818 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚5749 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53262366 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10076653 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192595519956992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6147236752896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45584816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53262366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22461231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚4900.116619741 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6079851216384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837191 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12384578 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6070573841990 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198742674431488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875409999552070 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6079851216384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837191 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873832935501824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873832918462464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6080192562688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12384838 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53252375 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22461494 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10076656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198742775322112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879912769678848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚869339442749440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976É|AmÓ ÕíM _step‚472 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873832935501824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192595538569216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6147154474496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45584818 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53252375 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837191 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12384578 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6070573841990 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198742674431488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875409999552070 UEDeviceStatsMonitor.on_train_batch_end/requested_byteݦn s.large_pool.peak‚ 23025176576 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6079851216384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837191 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873832935501824 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873832918462464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6080192562688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12384838 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53252375 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22461494 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10076656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198742775322112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879912769678848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚869339442749440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6080192562688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45584818 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚5749 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53262366 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10076653 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192595519956992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6147236752896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45584816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53262366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22461231 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚4900.116619741 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6079851216384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194233547737088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53707752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6120519825066 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6129534091264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53697775 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚5799 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656559 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45958786 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22649423 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200431056368128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876518465671168 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45958784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12487262 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚882638627906984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚887177621384192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚881047728914432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6197609528832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6120179275176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚882638985496234 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53697775 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚4939.460619511 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚881047745953792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6129875430400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12486991 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10162158 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887177263005696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887177263005696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194233547737088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53707752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6120519825066 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6129534091264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53697775 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚5799 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656559 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45958786 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22649423 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200431056368128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876518465671168 µM£2L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45958784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12487262 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚882638627906984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚887177621384192 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚881047728914432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6197609528832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6120179275176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274ۂû2å- _wandbruntime‚4940  _runtime‚4940-š ±î&æèM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194266976495616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚882786149632578 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53706827 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12489109 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673243 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45966418 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚881194969944064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚876664976343040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚887325516662272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53716818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6198556575744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6130888064512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10163906 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200465514459136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194266957883392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6198638854144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚4940.071703646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53716818 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚881194986983424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10163903 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚882786507221940 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673243 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚881194969944064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6130546718208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200465615349760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚876664959303680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6121190328898 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 % _timestamp‚1.7842777479458435e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22653012P®"Ëó&çíM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45966416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45966418 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚881194986983424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12489361 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22653012 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194266976495616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚882786149632578 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53706827 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12489109 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673243 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45966418 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚881194969944064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚876664976343040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚887325516662272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53716818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6198556575744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6130888064512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10163906 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200465514459136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194266957883392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6198638854144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚4940.071703646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53716818 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚881194986983424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10163903 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚882786507221940 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673243 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚881194969944064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6130546718208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200465615349760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚876664959303680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6121190328898 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 % _timestamp‚1.7842777479458435e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894590009989120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888409780396032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54162204 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890015135576742 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10249408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6180570932224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202153847135744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894590368367616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46340384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195904967051264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502590 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54152227 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46340386 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202153950130688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195904985663488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54152227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12592041 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚479 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180229593088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12591774 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22841452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚883843982225408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890014777987492 F@DeviceStatsMonitor.on_traž¼‘_ in_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚883843999264768 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22841182 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180229593088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6171136311974 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888409797435392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46340386 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54162204 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6248964467200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888409780396032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46340384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202153950130688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195904985663488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54152227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12592041 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚479 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180229593088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12591774 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22841452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚883843982225408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890014777987492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚883843999264768 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22841182 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180229593088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6171136311974 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888409797435392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46340386 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54162204 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6248964467200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888409780396032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46340384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894590009989120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888409780396032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54162204 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890015135576742 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10249408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6180570932224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202153847135744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894590368367616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46340384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195904967051264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502590 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54152227 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46340386 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976ߪ¾¥ žõ‚(:“ ›ÖçÒŸ¼½¥ gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Ôºö‚(:ö ›ÖçÒèòð§) disk./.usagePercent‚36.61770766634837 disk.sda.in‚0( proc.memory.availableMB‚ 496.15234375 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd3.in‚0 network.sent‚ 9.263052e+06 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 255.5546875 disk.nvme0n1.in‚ 255.5546875 cpu‚26.007059949726752 disk.drbd2165.in‚0 network.recv‚ 1.327166e+06( disk.nvme0c0n1.out‚26328.06201171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 167.5390625 disk.dm-0.in‚ 255.5546875 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 167.5390625 disk.rbd1.out‚95.91943359375 disk.rbd3.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2808.62890625* proc.memory.percent‚17.142510414123535$ disk./.usageGB‚545.6303062438965 disk.rbd0.out‚0 disk.rbd1.in‚ 0.01171875# memory_percent‚96.9717264175415 disk.sda.out‚ 167.5390625# disk.dm-0.out‚26328.06201171875& disk.nvme0n1.out‚26328.06201171875 disk.sda1.out‚0YÂ}2÷- _wandbruntime‚4980  _runtime‚4980àX\ò&øìM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100519288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚894738263645696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12593858 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12594119 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚883990492897280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6181242220032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚4980.057628062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6181583566336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54171270 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888557021425664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195938414422016 trainer/global_step‚5849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6181583566336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54171270 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888557021425664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202188284497920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46348016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6249888688128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚890162657302448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚890162299713086 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46348018 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195938395809792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46348018 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894738622031360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10251156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22845011 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202188402165760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54161279 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22845275 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888557038465024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894738622031360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0bÉä×ùñM BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10251156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22845011 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202188402165760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54161279 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22845275 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888557038465024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894738622031360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100519288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚894738263645696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12593858 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12594119 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚883990492897280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6181242220032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚4980.057628062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6181583566336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54171270 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888557021425664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195938414422016 trainer/global_step‚5849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚480 YGDeviceStatsMonitor.on_train_batch_end/actPÇá ive_bytes.small_pool.allocated‚ 6181583566336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54171270 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888557021425664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202188284497920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46348016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6249888688128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚890162657302448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚890162299713086 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46348018 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195938395809792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46348018 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894738622031360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12696263 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54616656 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚895771831877632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚5019.155012762 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23032663 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚891169515819008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231266434048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46721984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚483 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6300120697856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚902003115351040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46721986 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54606679 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚902002756972544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚895771848916992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12696005 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚891169532858368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6221752798882 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46721986 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚895771848916992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203876525675520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚902003115351040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197576423589888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231266434048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚895771831877632 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902002756972544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10336658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897391285657250 trainer/global_step‚5899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10336661 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203876626573312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6230925094912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328663 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54616656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197576404977664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221412248992 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6300202983424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438Ìì_Øë'†åO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231266434048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚895771831877632 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902002756972544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10336658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897391285657250 trainer/global_step‚5899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10336661 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203876626573312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6230925094912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328663 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54616656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197576404977664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221412248992 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6300202983424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23032924 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 # _timestamp‚1.78427782702925e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54606679 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12696263 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54616656 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚895771831877632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚5019.155012762 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23032663 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚891169515819008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231266434048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46721984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚483 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6300120697856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚902003115351040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46721986 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54606679 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚902002756972544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚895771848916992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12696005 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚891169532858368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6221752798882 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46721986 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚895771848916992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203876525675520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚902003115351040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197576423589888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976ã{`QU2‡- _wandbruntime‚5019  _runtime‚5019>RTˆëM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203910976498176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54625722 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.larïÖ´½ge_pool.allocated‚10338406 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10338403 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚897538449793594 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚891316026490880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895919089946624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚895919072907264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚902151369014784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚895919072907264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101345347 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54625722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897538807382956 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6222763852716 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46729618 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54615731 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6231937721856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232279068160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46729616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚5899 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197609833736192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232279068160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12698340 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12698082 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 $ _timestamp‚1.784277827640535e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23036485 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚484 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101355340Ü+YNö&‰ðM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54625722 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10338406 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10338403 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚897538449793594 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚891316026490880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895919089946624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚895919072907264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚902151369014784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚895919072907264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101345347 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54625722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897538807382956 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6222763852716 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46729618 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54615731 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6231937721856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232279068160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46729616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚5899 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197609833736192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232279068160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12698340 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12698082 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 $ _timestamp‚1.784277827640535e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23036485 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚484 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101355340 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203910976498176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088äù”+2Š- _wandbruntime‚5019  _runtime‚50192HxV‹þ 0loss/FRt2/assignment_loss‚0.9686669326043343 1loss/FBt2/assignment_loss‚0.05531868351787583 /loss/FRt2/detection_loss‚1.2032990545182287 1loss/SRqqt2/detection_loss‚0.6093301486429613 /loss/FBt2/detection_loss‚0.7862697507565221 trainer/global_step‚5899 1loss/FBt1/assignment_loss‚0.03094738281029217 /loss/FRt1/detection_loss‚0.5435598491969865 /loss/FBt1/detection_loss‚0.4582508208440875 /loss/FRt1/assignment_loss‚1.778145172960682 $loss/total_loss‚8.44965400715077 epoch‚14 % _timestamp‚1.7842778276446683e+09 0loss/SRqqt1/assignment_loss‚0.52289551416834 2loss/SRqqt2/assignment_loss‚0.5003835380608946 1loss/SRqqt1/detection_loss‚0.9925871590695654 _runtime‚5019.770141386 _step‚485_@ Œƒ 0loss/SRqqt1/assignment_loss‚0.52289551416834 2loss/SRqqt2/assignment_loss‚0.5003835380608946 1loss/SRqqt1/detection_loss‚0.9925871590695654 _runtime‚5019.770141386 _step‚485 0loss/FRt2/assignment_loss‚0.9686669326043343 1loss/FBt2/assignment_loss‚0.05531868351787583 /loss/FRt2/detection_loss‚1.2032990545182287 1loss/SRqqt2/detection_loss‚0.6093301486429613 /loss/FBt2/detection_loss‚0.7862697507565221 trainer/global_step‚5899 1loss/FBt1/assignment_loss‚0.03094738281029217 /loss/FRt1/detection_loss‚0.5435598491969865 /loss/FBt1/detection_loss‚0.4582508208440875 /loss/FRt1/assignment_loss‚1.778145172960682 $loss/total_loss‚8.44965400715077 epoch‚14 % _timestamp‚1.7842778276446683e+09åCŒ•?ž‚(:“ ÈÖçÒ×Ë«£ gpu.0.gpu‚70.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚284.867' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.973400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05Š„°Ž‚(:÷ ÈÖçÒ—ˆ¯¦) disk./.usagePercent‚36.61745882702509 disk.dm-1.in‚0 disk.rbd1.out‚95.98193359375$ memory_percent‚96.97256088256836& disk.nvme0n1.out‚26370.06103515625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 257.09765625 disk.rbd0.out‚0 disk.sda.out‚ 174.4453125& proc.memory.availableMB‚ 496.015625* proc.memory.percent‚17.142510414123535 proc.cpu.threads‚17 network.recv‚ 1.33844e+06( disk.nvme0c0n1.out‚26370.06103515625 disk.dm-1.out‚ 174.4453125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚26370.06103515625 disk.sda2.out‚0 disk.sda3.in‚0 cpu‚26.004499071080964 network.sent‚ 9.372588e+06 disk.nvme0n1.in‚ 257.09765625" disk.nvme0c0n1.in‚ 257.09765625 disk.sda3.out‚ 174.4453125# proc.memory.rssMB‚ 2808.62890625$ disk./.usageGB‚545.6265983581543 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875!ÐTJ‚(:„ ×ÖçÒØÜߤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PÛ•ú‚(:ï ×ÖçÒ×Ȧ disk.drbd2165.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2808.62890625* proc.memory.percent‚17.142510414123535 disk.rbd1.out‚96.04443359375 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 174.97265625 disk.sda3.out‚ 174.97265625 cpu‚26.003598751414312 proc.cpu.threads‚17) disk./.usagePercent‚36.61756967828742 disk.rbd3.out‚0 disk.dm-0.in‚257.8125# disk.dm-0.out‚26382.32470703125& disk.nvme0n1.out‚26382.32470703125 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚96.97117805480957 network.sent‚ 9.435489e+06 network.recv‚ 1.34309e+06 disk.dm-1.out‚ 174.97265625 disk.sda3.in‚0' proc.memory.availableMB‚ 496.2421875$ disk./.usageGB‚545.6282501220703 disk.sda2.out‚0 disk.nvme0c0n1.in‚257.8125 disk.nvme0n1.in‚257.8125( disk.nvme0c0n1.out‚26382.32470703125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0³å%ž‘‚(:“ æÖçÒØç¤ gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚148.601' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.720200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÐüÞü’‚(:ñ æÖçÒÈûµ¦$ memory_percent‚96.97117805480957 network.recv‚ 1.346446e+06 disk.nvme0n1.in‚ 258.27734375 disk.rbd0.out‚0' proc.memory.availableMB‚ 496.2421875 cpu‚26.00280542803312 network.sent‚ 9.442637e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 proc.cpu.threads‚17 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 258.27734375( disk.nvme0c0n1.out‚26387.72412109375 disk.sda2.out‚0 disk.rbd1.out‚96.04443359375 disk.rbd3.out‚0( disk./.usagePercent‚36.6175950230333 disk.sda.in‚0# proc.memory.rssMB‚ 2808.62890625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 175.53125 disk.sda3.out‚ 175.53125 disk.dm-1.in‚0 disk.dm-0.in‚ 258.27734375* proc.memory.percent‚17.142510414123535$ disk./.usageGB‚545.6286277770996 disk.sda1.in‚0 disk.dm-1.out‚ 175.53125# disk.dm-0.out‚26387.72412109375& disk.nvme0n1.out‚26387.72412109375 disk.sda1.out‚0 disk.rbd3.in‚06ˈë2“- _wandbruntime‚5059  _runtime‚5059-áÎÆ”À & lr-AdamW/pg1‚0.0003851915126149932 & lr-AdamW/pg2‚0.0003851915126149932 trainer/global_step‚5949 $ _timestamp‚1.784277866997213e+09 _runtime‚5059.122794374 _step‚4864§Ë•Å & lr-AdamW/pg2‚0.0003851915126149932 trainer/global_step‚5949 $ _timestamp‚1.784277866997213e+09 _runtime‚5059.122794374 _step‚486 & lr-AdamW/pg1‚0.0003851915126149932æªR#42–- _wandbruntime‚5059  _runtime‚50592% æ'—àO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12800350 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6351421535744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚898495066451968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903133900398592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6272028735900 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚909415862334464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6281620596736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12800608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚487 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6281961935872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5949 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55071108 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚909415862334464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47103584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205599375823360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6351514307072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199247842904064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚904767435737758 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784277866998256e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6281961935872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55061131 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199247861516288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6272369285790 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47103586 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903133900398592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6281620596736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23224519 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚903133883359232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10423908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚904767078148508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23224258 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909415503955968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205599264439808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚909415503955968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55061131 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47103584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47103586 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909415503955968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205599264439808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚909415503955968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55061131 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47103584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47103586 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6281620596736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12800608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚487 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6281961935872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5949 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55071108 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚909415862334464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47103584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205599375823360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6351514307072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199247842904064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚904767435737758 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784277866998256e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6281961935872çOþ®2™- _wandbruntime‚5059  _runtime‚5059ôöóò&šìM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6282974569984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23228274 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205633721622528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898641560084480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903281141428224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47111216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903281141428224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6282633223680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909564115998208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199281271662592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23227980 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6273380339624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55070183 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10425653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚904914599874102 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6282974569984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12802327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205633828804608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909563757612544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47111216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6273039789622 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088Î/¿m÷&›ñM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚909564115998208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚904914599874102 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6282974569984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12802327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205633828804608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909563757612544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47111216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6273039789622 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6282974569984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23228274 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205633721622528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898641560084480 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903281141428224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47111216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903281141428224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6282633223680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909564115998208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199281271662592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23227980 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6273380339624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55070183 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10425653 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916828250939392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6332657437696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47485186 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6332316098560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6332657437696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200919299442688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55525560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚912143585818266 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6322985772698 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚916828609317888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23415240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47485186 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚910495951880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916828250939392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6322645222808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10511161 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103010746 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905820600045568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47485184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23415513 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55515583 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910495934840832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6402735401984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905820600045568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47485184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23415513 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55515583 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910495934840832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6402735401984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916828250939392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6332657437696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47485186 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6332316098560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6332657437696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200919299442688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55525560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚912143585818266 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6322985772698 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚916828609317888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23415240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47485186 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚910495951880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916828250939392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6322645222808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10511161 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103010746 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚910643192909824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55534626 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6333328725504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6403756727808 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23419328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6403847394816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47492818 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55524635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10512903 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910643175870464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916976862981632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚916976862981632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905967110717440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55524635 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916976504595968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23419044 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10512906 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6323656276530 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842779075410347e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103017451 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6323996826532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017451 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6333670071808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚207356575596032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚5099.666810495öe£&«ñM B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚5099.666810495 N>DeviceStatsMonitor.on_train_batch_end/allocated_byte…¨Û~Ñ%s.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚910643192909824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55534626 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6333328725504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6403756727808 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23419328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6403847394816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47492818 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55524635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10512903 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910643175870464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916976862981632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚916976862981632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905967110717440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55524635 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916976504595968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23419044 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10512906 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6323656276530 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842779075410347e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103017451 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6323996826532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017451 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6333670071808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚207356575596032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ìÔG ¹2¬- _wandbruntime‚5099  _runtime‚5099šYN› ­ƒ /loss/FRt1/detection_loss‚0.5013362906194267 1loss/SRqqt2/detection_loss‚0.6482091376670577 trainer/global_step‚5999 _step‚493 2loss/SRqqt1/assignment_loss‚0.5307766476337312 2loss/SRqqt2/assignment_loss‚0.5172808207505465 1loss/FBt2/assignment_loss‚0.10070668062589253 /loss/FBt1/detection_loss‚0.4122824567927729 epoch‚14 % _timestamp‚1.7842779075441475e+09 0loss/FRt1/assignment_loss‚1.7785254892220166 2loss/FBt1/assignment_loss‚0.023201169502997807 /loss/FRt2/detection_loss‚1.3731313025536172 1loss/SRqqt1/detection_loss‚1.0012034651296942 _runtime‚5099.669472612 0loss/FRt2/assignment_loss‚1.0028857830592564 /loss/FBt2/detection_loss‚1.0792810719659818 %loss/total_loss‚8.968820315522992n6<𮈠0loss/FRt2/assignment_loss‚1.0028857830592564 /loss/FBt2/detection_loss‚1.0792810719659818 %loss/total_loss‚8.968820315522992 /loss/FRt1/detection_loss‚0.5013362906194267 1loss/SRqqt2/detection_loss‚0.6482091376670577 trainer/global_step‚5999 _step‚493 2loss/SRqqt1/assignment_loss‚0.5307766476337312 2loss/SRqqt2/assignment_loss‚0.5172808207505465 1loss/FBt2/assignment_loss‚0.10070668062589253 /loss/FBt1/detection_loss‚0.4122824567927729 epoch‚14 % _timestamp‚1.7842779075441475e+09 0loss/FRt1/assignment_loss‚1.7785254892220166 2loss/FBt1/assignment_loss‚0.023201169502997807 /loss/FRt2/detection_loss‚1.3731313025536172 1loss/SRqqt1/detection_loss‚1.0012034651296942 _runtime‚5099.669472612퀱“¯‚(:ˆ “×çÒ‡ˆ›¥ gpu.0.gpu‚80.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þø‚Jû°‚(:ð “×çÒž­£§* proc.memory.percent‚17.142510414123535 network.recv‚ 1.358199e+06% disk.nvme0n1.out‚26413.7099609375 disk.sda2.out‚0 disk.dm-1.in‚0 network.sent‚ 9.526875e+06 disk.sda1.in‚0 disk.dm-1.out‚ 177.09375 cpu‚26.000391146703603 proc.cpu.threads‚17) disk./.usagePercent‚36.61737178446345 disk.rbd3.in‚0" disk.dm-0.out‚26413.7099609375" disk.nvme0c0n1.in‚ 260.18359375 disk.sda.out‚ 177.09375 disk.sda3.out‚ 177.09375 disk.rbd3.out‚0 disk.dm-0.in‚ 260.18359375# disk./.usageGB‚545.625301361084 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚96.97496891021729# proc.memory.rssMB‚ 2808.62890625' disk.nvme0c0n1.out‚26413.7099609375 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚96.16943359375( proc.memory.availableMB‚ 495.62109375 disk.nvme0n1.in‚ 260.18359375 disk.sda1.out‚0 disk.rbd0.in‚0äž±‚(:“ ¢×çÒå鉥 gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.949' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.989800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…þû‚ÿ²‚(:ô ¢×çÒºÓÀ§ disk.drbd2165.in‚0 disk.dm-1.out‚ 177.23046875 disk.sda1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.142510414123535 disk.sda.out‚ 177.23046875 disk.sda3.out‚ 177.23046875# proc.memory.rssMB‚ 2808.62890625 proc.cpu.threads‚17 network.recv‚ 1.362643e+06$ disk./.usageGB‚545.6252212524414 disk.nvme0n1.in‚ 260.8984375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 260.8984375 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 260.8984375' disk.nvme0c0n1.out‚26420.5908203125 disk.rbd1.in‚ 0.01171875$ memory_percent‚96.97513580322266 network.sent‚ 9.578926e+06 disk.rbd1.out‚96.16943359375 disk.rbd3.out‚0 cpu‚26.011239293630222) disk./.usagePercent‚36.61736640830523" disk.dm-0.out‚26420.5908203125 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 495.59375% disk.nvme0n1.out‚26420.5908203125u×Ù2³- _wandbruntime‚5116  _runtime‚5116BR{{¡=´›{ 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7092832326889038 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48898226022720337 *jet/accuracy_1_of_3‚0.9291855096817017 % _timestamp‚1.7842779246424503e+09 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.665340006351471 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5032687783241272 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30204853415489197 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.92240971326828 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025895535945892334 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.715456485748291 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2936077415943146 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026566144078969955 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4011181890964508 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8092592358589172 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6781307458877563 )jet/accuracy_1_of_1‚0.365614116191864 *jet/accuracy_2_of_4‚0.8040772676467896 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7534235715866089 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 (particle/accuracy‚0.8460712432861328 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8619723320007324 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6280192732810974 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04486674442887306 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962906837463379 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.026566144078969955 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6227286458015442 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03875511512160301 particle/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6692719459533691 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9959307312965393 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22726374864578247 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5885416269302368 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25751563906669617 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 *jet/accuracy_1_of_2‚0.8036556243896484 *jet/accuracy_3_of_3‚0.0708862915635109 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929646849632263 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906299114227295 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5555232763290405 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984418749809265 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6908400654792786 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8520188331604004 particle/accuracy_1_of_1‚1 trainer/global_step‚5999 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 jet/accuracy_5_of_5‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.297654279929308 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4014511704444885 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.755433201789856 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7616890668869019 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7643503546714783 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24561898410320282 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24632905423641205 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6016920804977417 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3129817843437195 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 epoch‚14 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.783729076385498 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_4‚0.09480094164609909 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666865348816 _step‚494 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6731910705566406 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5761373043060303 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8179516792297363 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_2‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21933771669864655 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22341680526733398 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.229871928691864 /particle/accuracy_3_of_4‚0.9936224818229675 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5379166007041931 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969077110290527 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5882290005683899 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9889675974845886 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8115457892417908 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6361430287361145 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6434139609336853 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6555609703063965 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49781981110572815 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7516940832138062 *jet/accuracy_2_of_2‚0.1614374816417694 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917799830436707 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.055497169494628906 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024645118042826653 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5057557225227356 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956583976745605 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.71516352891922 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2904396057128906 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024907801300287247 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8001324534416199 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7995349168777466 7"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.67107093334198 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7571223378181458 'particle/f_score‚0.6368451118469238 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6283863186836243 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12101267278194427 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13166667520999908 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8984508514404297 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13620446622371674 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7487888932228088 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9237552881240845 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6732465028762817 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1537516564130783 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6951459050178528 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929404258728027 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6991232633590698 /particle/accuracy_4_of_4‚0.9503598213195801 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8623452186584473 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40027478337287903 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960535168647766 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13817837834358215 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3133934736251831 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.044036511331796646 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3034903407096863 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23827730119228363 particle/accuracy_1_of_2‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9892956614494324 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24905872344970703 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7213529944419861 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.789720356464386 *jet/accuracy_1_of_4‚0.9691979289054871 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7554454803466797 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22428710758686066 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25995171070098877 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3555452823638916 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3034903407096863 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8520188331604004 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.71955406665802 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2861102521419525 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49768516421318054 ,jet/accuracy_4_of_4‚0.006769374944269657 +particle/specificity‚0.9055652618408203 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7430376410484314 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7922524213790894 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.789341151714325 particle/accuracy_2_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3336777985095978 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43863487243652344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _runtime‚5116.768692657 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.366310715675354 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5656370520591736 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.488762229681015 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7402100563049316 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07458104193210602 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983088374137878 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08509749919176102 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29935237765312195 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31218457221984863 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7062381505966187 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955142736434937 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23861737549304962 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026102792471647263 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6732465028762817 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6285319924354553ŠpÿW› µ { =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04486674442887306 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962906837463379 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.026566144078969955 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6227286458015442 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03875511512160301 particle/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6692719459533691 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9959307312965393 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22726374864578247 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5885416269302368 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25751563906669617 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 *jet/accuracy_1_of_2‚0.8036556243896484 *jet/accuracy_3_of_3‚0.0708862915635109 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929646849632263 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906299114227295 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5555232763290405 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984418749809265 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6908400654792786 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8520188331604004 particle/accuracy_1_of_1‚1 trainer/global_step‚5999 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 jet/accuracy_5_of_5‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.297654279929308 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4014511704444885 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.755433201789856 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7616890668869019 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.76435035‹ÃT¥ 346714783 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24561898410320282 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24632905423641205 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6016920804977417 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3129817843437195 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 epoch‚14 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.783729076385498 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_4‚0.09480094164609909 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666865348816 _step‚494 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6731910705566406 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5761373043060303 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8179516792297363 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_2‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21933771669864655 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22341680526733398 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.229871928691864 /particle/accuracy_3_of_4‚0.9936224818229675 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5379166007041931 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969077110290527 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5882290005683899 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9889675974845886 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8115457892417908 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6361430287361145 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6434139609336853 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6555609703063965 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49781981110572815 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7516940832138062 *jet/accuracy_2_of_2‚0.1614374816417694 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917799830436707 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.055497169494628906 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024645118042826653 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5057557225227356 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956583976745605 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.71516352891922 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2904396057128906 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024907801300287247 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8001324534416199 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7995349168777466 7"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.67107093334198 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7571223378181458 'particle/f_score‚0.6368451118469238 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6283863186836243 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12101267278194427 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13166667520999908 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8984508514404297 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13620446622371674 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7487888932228088 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9237552881240845 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6732465028762817 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1537516564130783 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6951459050178528 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929404258728027 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6991232633590698 /particle/accuracy_4_of_4‚0.9503598213195801 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8623452186584473 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40027478337287903 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960535168647766 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13817837834358215 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3133934736251831 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.044036511331796646 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3034903407096863 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23827730119228363 particle/accuracy_1_of_2‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9892956614494324 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24905872344970703 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7213529944419861 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.789720356464386 *jet/accuracy_1_of_4‚0.9691979289054871 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7554454803466797 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22428710758686066 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25995171070098877 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3555452823638916 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3034903407096863 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8520188331604004 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.71955406665802 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2861102521419525 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49768516421318054 ,jet/accuracy_4_of_4‚0.006769374944269657 +particle/specificity‚0.9055652618408203 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7430376410484314 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7922524213790894 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.789341151714325 particle/accuracy_2_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3336777985095978 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43863487243652344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _runtime‚5116.768692657 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.366310715675354 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5656370520591736 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.488762229681015 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7402100563049316 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07458104193210602 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983088374137878 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08509749919176102 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29935237765312195 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31218457221984863 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7062381505966187 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955142736434937 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23861737549304962 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.026102792471647263 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6732465028762817 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.6285319924354553 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7092832326889038 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48898226022720337 *jet/accuracy_1_of_3‚0.9291855096817017 % _timestamp‚1.7842779246424503e+09 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.665340006351471 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5032687783241272 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30204853415489197 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.92240971326828 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025895535945892334 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.715456485748291 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2936077415943146 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026566144078969955 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4011181890964508 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8092592358589172 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6781307458877563 )jet/accuracy_1_of_1‚0.365614116191864 *jet/accuracy_2_of_4‚0.8040772676467896 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7534235715866089 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 (particle/accuracy‚0.8460712432861328 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8619723320007324 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6280192732810974 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484îSŽi,¶‚(:’ ±×çÒŽúŠ§ gpu.0.gpu‚86.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.569' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.71379999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åþNÛõ·‚(:ê ±×çÒ莑ª# proc.memory.rssMB‚ 2808.62890625 network.recv‚ 1.366229e+06& disk.nvme0c0n1.out‚26681.884765625 disk.rbd3.in‚0# disk./.usageGB‚545.638053894043 cpu‚26.010937373827865 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 260.8984375$ memory_percent‚95.47545909881592( proc.memory.availableMB‚ 741.30078125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0) disk./.usagePercent‚36.61822761765041 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.142510414123535 disk.dm-1.out‚ 179.40625 disk.rbd1.out‚96.23193359375 disk.rbd3.out‚0 disk.dm-0.in‚ 260.8984375 network.sent‚ 9.581949e+06 disk.nvme0n1.in‚ 260.8984375$ disk.nvme0n1.out‚26681.884765625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 179.40625! disk.dm-0.out‚26681.884765625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 179.40625Ö“¿œ¸‚(:‘ À×çÒÈÙܤ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.78' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÊ a¹‚(:ö À×çÒà›§ disk.dm-1.out‚ 180.22265625! disk.nvme0c0n1.in‚ 261.7265625 disk.sda.in‚0# proc.memory.rssMB‚ 2808.62890625 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 180.22265625' disk.nvme0c0n1.out‚26945.5458984375 disk.rbd3.out‚0 network.recv‚ 1.370865e+06 disk.sda2.out‚0 disk.sda3.out‚ 180.22265625 network.sent‚ 9.641663e+06 disk.sda2.in‚0" disk.dm-0.out‚26945.5458984375 cpu‚26.01007170988635 disk.rbd0.in‚0$ memory_percent‚95.47622203826904( proc.memory.availableMB‚ 741.17578125* proc.memory.percent‚17.142510414123535 disk.nvme0n1.in‚ 261.7265625 disk.sda1.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚26945.5458984375 disk.sda1.out‚0) disk./.usagePercent‚36.61817488009836 disk.sda3.in‚0 disk.rbd1.out‚96.23193359375 disk.rbd3.in‚0$ disk./.usageGB‚545.6372680664062 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 261.7265625 proc.cpu.threads‚17§2³+2º- _wandbruntime‚5158  _runtime‚5158þàÓÇ»Á trainer/global_step‚6049 % _timestamp‚1.7842779660762708e+09 _runtime‚5158.201903852 _step‚495 & lr-AdamW/pg1‚0.0003822302754811417 & lr-AdamW/pg2‚0.0003822302754811417Ú] þÌ¼Æ & lr-AdamW/pg1‚0.0003822302754811417 & lr-AdamW/pg2‚0.0003822302754811417 trainer/global_step‚6049 % _timestamp‚1.7842779660762708e+09 _runtime‚5158.201903852 _step‚495ï\óÙ12½- _wandbruntime‚5158  _runtime‚5158¬2’ç'¾áO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47987450 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56050339 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚926188656159744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56050339 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47987452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚926188656159744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13040083 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚921446118838088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6472130017792 % _timestamp‚1.7842779660775442e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6390981434950 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6391321984840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6401086147584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚919787552972800 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚921445761248838 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202606630856704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56040362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚926188297781248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚915054779813888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401086147584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23641011 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚6049 _step‚496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚919787570012160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202606649468928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47987452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209078855480832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104027812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚926188297781248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚5158.203459778 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56040362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47987450 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209078760874496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10600928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚919787552972800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888ì:– ›¿æO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚6049 _step‚496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚919787570012160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202606649468928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47987452 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209078855480832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104027812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚926188297781248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚5158.203459778 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56040362 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47987450 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209078760874496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10600928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚919787552972800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47987450 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56050339 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚926188656159744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56050339 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47987452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚926188656159744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13040083 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚75994112 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚921446118838088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6472130017792 % _timestamp‚1.7842779660775442e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6390981434950 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6391321984840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6401086147584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚919787552972800 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚921445761248838 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202606630856704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56040362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚926188297781248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚915054779813888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401086147584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23641011 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ð/H22À- _wandbruntime‚5158  _runtime‚5158‚zRªñ&ÁëM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6473239929856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚921593640563794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47995082 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104054489 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13042472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _step‚497 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚926336909823488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23645148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚919934794002432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202640059615232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6473155554304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚919934811041792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56049414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104054489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10602673 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47995084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13042216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6401757435392 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23644889 $ _timestamp‚1.784277966689899e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104044496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚926336551437824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10602676 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209113318157312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚919934811041792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209113215169536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6391992488672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6401757435392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402098781696 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚915201307525120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚5158.815696079 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23644889 $ _timestamp‚1.784277966689899e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104044496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚926336551437824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10602676 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209113318157312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚919934811041792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209113215169536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6391992488672 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6401757435392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402098781696 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚915201307525120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚5158.815696079 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47995082 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104054489 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13042472 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _step‚497 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚926336909823488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23645148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚919934794002432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202640059615232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6473155554304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚919934811041792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56049414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104054489 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10602673 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47995084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13042216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169ñ[iÑx2Ã- _wandbruntime‚5158  _runtime‚5158úœùz Ć /loss/FBt2/detection_loss‚0.7103363920879715 0loss/FRt2/assignment_loss‚0.9456738702841659 /loss/FRt2/detection_loss‚1.2867133602601608 epoch‚15 trainer/global_step‚6049 _runtime‚5158.819031975 2loss/SRqqt2/assignment_loss‚0.5198520057671014 1loss/SRqqt1/detection_loss‚0.9682035285686692 _step‚498 0loss/FRt1/assignment_loss‚1.7457153674647294 2loss/SRqqt1/assignment_loss‚0.5077882715528597 /loss/FRt1/detection_loss‚0.5000974054055425 1loss/SRqqt2/detection_loss‚0.5796529115551988 %loss/total_loss‚8.159203459363045 % _timestamp‚1.7842779666937068e+09 2loss/FBt1/assignment_loss‚0.031696273005795486 3loss/FBt2/assignment_loss‚0.0030604753827958575 0loss/FBt1/detection_loss‚0.36041359802805567¦rÅ‹ trainer/global_step‚6049 _runtime‚5158.819031975 2loss/SRqqt2/assignment_loss‚0.5198520057671014 1loss/SRqqt1/detection_loss‚0.9682035285686692 _step‚498 0loss/FRt1/assignment_loss‚1.7457153674647294 2loss/SRqqt1/assignment_loss‚0.5077882715528597 /loss/FRt1/detection_loss‚0.5000974054055425 1loss/SRqqt2/detection_loss‚0.5796529115551988 %loss/total_loss‚8.159203459363045 % _timestamp‚1.7842779666937068e+09 2loss/FBt1/assignment_loss‚0.031696273005795486 3loss/FBt2/assignment_loss‚0.0030604753827958575 0loss/FBt1/detection_loss‚0.36041359802805567 /loss/FBt2/detection_loss‚0.7103363920879715 0loss/FRt2/assignment_loss‚0.9456738702841659 /loss/FRt2/detection_loss‚1.2867133602601608 epoch‚15òwÉú-œÆ‚(:‘ Ï×çÒù‰Ù¥ gpu.0.gpu‚15.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.52380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0} FƒÇ‚(:ü Ï×çÒ¶Ïߧ disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 262.44140625 disk.sda1.out‚0 disk.sda2.in‚0 network.sent‚ 9.657455e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 183.50390625# memory_percent‚95.4763650894165# proc.memory.rssMB‚ 2808.62890625( disk.nvme0c0n1.out‚26963.92626953125) disk./.usagePercent‚36.61795880973946 disk.sda.in‚0 disk.sda3.out‚ 183.50390625 cpu‚26.009218438015978 proc.cpu.threads‚17 disk.sda3.in‚0$ disk./.usageGB‚545.6340484619141& disk.nvme0n1.out‚26963.92626953125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 262.44140625 disk.sda.out‚ 183.50390625 disk.rbd1.in‚ 0.01171875( proc.memory.availableMB‚ 741.15234375 disk.nvme0n1.in‚ 262.44140625 disk.rbd0.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.142510414123535 disk.sda1.in‚0 disk.drbd2165.in‚0 network.recv‚ 1.373714e+06 disk.rbd1.out‚96.29443359375# disk.dm-0.out‚26963.92626953125¥ ±žÈ‚(:“ Þ×çÒ”×æ¤ gpu.0.gpu‚49.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XäþüÉ‚(:ñ Þ×çÒËý’§ disk.dm-1.out‚ 184.53515625 disk.rbd3.in‚0) disk./.usagePercent‚36.61796469791275 disk.sda1.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.out‚26972.71484375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 263.26953125 disk.sda.in‚0 disk.dm-1.in‚0 cpu‚26.00843210561276 network.sent‚ 9.708844e+06$ disk./.usageGB‚545.6341361999512 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 network.recv‚ 1.377982e+06% disk.nvme0c0n1.out‚26972.71484375 disk.sda.out‚ 184.53515625 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.142510414123535 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd1.out‚96.29443359375 disk.dm-0.in‚ 263.26953125 proc.cpu.threads‚17 disk.sda3.out‚ 184.53515625# proc.memory.rssMB‚ 2808.62890625# memory_percent‚95.4765796661377' proc.memory.availableMB‚ 741.1171875 disk.nvme0n1.in‚ 263.26953125# disk.nvme0n1.out‚26972.71484375 disk.drbd2165.in‚0…‰ “Ê‚(:ˆ í×çÒ쩬¤ gpu.0.gpu‚50.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+qJ™ëË‚(:à í×çÒ›¢Ó¦ disk.sda3.in‚0 disk.rbd1.out‚96.35693359375 disk.dm-0.out‚26980.98828125 disk.sda.in‚0 disk.sda.out‚185.6875 disk.sda2.in‚0 disk.dm-0.in‚ 263.984375) disk./.usagePercent‚36.61776270796823 disk.dm-1.out‚185.6875 disk.rbd1.in‚ 0.01171875 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 263.984375 cpu‚26.007649080265566 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2808.62890625 network.sent‚ 9.713853e+06 disk.sda1.in‚0$ disk./.usageGB‚545.6311264038086 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 263.984375# disk.nvme0n1.out‚26980.98828125 disk.drbd2165.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚26980.98828125" memory_percent‚95.475172996521 proc.cpu.threads‚17 network.recv‚ 1.381372e+06 disk.sda3.out‚185.6875 disk.rbd3.out‚0( proc.memory.availableMB‚ 741.34765625* proc.memory.percent‚17.142510414123535Ïê´2Ì- _wandbruntime‚5197  _runtime‚5197÷KK‘ÉÍà % _timestamp‚1.7842780058340309e+09 _runtime‚5197.959661793 _step‚499 ' lr-AdamW/pg1‚0.00038073555325612553 ' lr-AdamW/pg2‚0.00038073555325612553 trainer/global_step‚6099È+íæÎÎÈ ' lr-AdamW/pg1‚0.00038073555325612553 ' lr-AdamW/pg2‚0.00038073555325612553 trainer/global_step‚6099 % _timestamp‚1.7842780058340309e+09 _runtime‚5197.959661793 _step‚499ó»[þ2Ï- _wandbruntime‚5197  _runtime‚5197‡wk_ÐáO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23832256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚933601044764672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚922380330446848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48369050 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6451781649408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204278068783104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚6099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23831981 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56494814 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚210801466350080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56504791 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56494814 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13144075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784278005835076e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚933601403143168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10688181 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204278087395328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6523378954752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚927149604454400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104873843 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48369052 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6441938471748 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚927149621493760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚927149604454400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10688178 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933601403143168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚928821911329346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚922380313407488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104863864 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933601044764672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13143803 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚210801352869376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚5002Ά³ì'ÑæO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933601403143168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚928821911329346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚922380313407488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104863864 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933601044764672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13143803 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚210801352869376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚500 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23832256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚933601044764672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚922380330446848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48369050 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6451781649408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204278068783104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚6099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23831981 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56494814 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚210801466350080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56504791 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56494814 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13144075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784278005835076e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚933601403143168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10688181 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204278087395328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6523378954752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚927149604454400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104873843 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48369052 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6441938471748 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚927149621493760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚927149604454400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10688178ôô؈ô2Ò- _wandbruntime‚5198  _runtime‚5198÷Šäò&ÓìM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13146135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13145871 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6442949525582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48376684 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933749656806912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚933749298421248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6452794283520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚928969433054940 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204311516153856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56503866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚210835911024640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚928969790644302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6442608975580 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23835794 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚927296862523392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104890541 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56503866 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚501 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚210835801745408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚927296845484032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56513857 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104880548 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚922526841118720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚922526824079360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10689923 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚927296845484032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6452452937216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚6099 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104880548 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48376682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204311497541632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 % _timestamp‚1.7842780064463549e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104890541 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚927296862523392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6452794283520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10689926 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48376684 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6524394870784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56513857 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6452452937216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48376682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23836061úuùÔñM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104890541 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚927296862523392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6452794283520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10689926 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48376684 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6524394870784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56513857 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6452452937216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48376682 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23836061 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13146135 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13145871 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6442949525582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48376684 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933749656806912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚933749298421248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6452794283520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚928969433054940 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204311516153856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56503866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚210835911024640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peÊû`^þak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚928969790644302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6442608975580 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23835794 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚927296862523392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104890541 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56503866 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚501 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚210835801745408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚927296845484032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56513857 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104880548 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚922526841118720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚922526824079360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10689923 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚927296845484032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6452452937216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚6099 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104880548 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48376682 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204311497541632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 % _timestamp‚1.7842780064463549e+09õâî2Õ- _wandbruntime‚5198  _runtime‚5198Ží²Ö‚ %loss/total_loss‚8.266527153829344 _step‚502 0loss/FRt2/assignment_loss‚0.9773491191734907 2loss/SRqqt1/assignment_loss‚0.4751968254038626 2loss/FBt1/assignment_loss‚0.005153215185624052 2loss/FBt2/assignment_loss‚0.003846059394034758 1loss/SRqqt1/detection_loss‚0.9492182169957617 /loss/FBt1/detection_loss‚0.6098289994267008 trainer/global_step‚6099 _runtime‚5198.574615637 /loss/FRt1/assignment_loss‚1.678390439386998 2loss/SRqqt2/assignment_loss‚0.5414150684137422 /loss/FBt2/detection_loss‚0.5805267785365383 % _timestamp‚1.7842780064492867e+09 epoch‚15 /loss/FRt2/detection_loss‚1.2446182585291528 0loss/SRqqt2/detection_loss‚0.640342230766546 /loss/FRt1/detection_loss‚0.5606419426168915‚1·S ׇ % _timestamp‚1.7842780064492867e+09 epoch‚15 /loss/FRt2/detection_loss‚1.2446182585291528 0loss/SRqqt2/detection_loss‚0.640342230766546 /loss/FRt1/detection_loss‚0.5606419426168915 %loss/total_loss‚8.266527153829344 _step‚502 0loss/FRt2/assignment_loss‚0.9773491191734907 2loss/SRqqt1/assignment_loss‚0.4751968254038626 2loss/FBt1/assignment_loss‚0.005153215185624052 2loss/FBt2/assignment_loss‚0.003846059394034758 1loss/SRqqt1/detection_loss‚0.9492182169957617 /loss/FBt1/detection_loss‚0.6098289994267008 trainer/global_step‚6099 _runtime‚5198.574615637 /loss/FRt1/assignment_loss‚1.678390439386998 2loss/SRqqt2/assignment_loss‚0.5414150684137422 /loss/FBt2/detection_loss‚0.5805267785365383ö”±“Ø‚(:ˆ ü×çÒñ̆¥ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;1‘2ýÙ‚(:ò ü×çÒ‚®¦§* proc.memory.percent‚17.142510414123535 disk.dm-1.out‚ 185.9453125( disk.nvme0c0n1.out‚26988.31494140625 disk.sda3.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2808.62890625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 185.9453125# disk.dm-0.out‚26988.31494140625 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 263.984375) disk./.usagePercent‚36.61777090020933 disk.sda.out‚ 185.9453125 disk.sda1.out‚0$ memory_percent‚95.47677040100098' proc.memory.availableMB‚ 741.0859375 disk.rbd0.in‚0 network.sent‚ 9.716817e+06 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 263.984375 disk.sda1.in‚0 disk.rbd1.out‚96.35693359375$ disk./.usageGB‚545.6312484741211 disk.nvme0c0n1.in‚ 263.984375 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 cpu‚26.00681107754544 network.recv‚ 1.384661e+06& disk.nvme0n1.out‚26988.31494140625·y ÆžÚ‚(:“ ‹ØçÒÓ”¯¤ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éëÖÛóÛ‚(:è ‹ØçÒķܦ# disk.nvme0n1.out‚26995.37109375 disk.sda.out‚ 186.24609375 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 proc.cpu.threads‚17 network.sent‚ 9.781796e+06 disk.rbd3.out‚0# proc.memory.rssMB‚ 2808.62890625 disk.sda3.in‚0( proc.memory.availableMB‚ 741.14453125 network.recv‚ 1.389184e+06$ memory_percent‚95.47641277313232 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 186.24609375* proc.memory.percent‚17.142510414123535 cpu‚26.005990528185805 disk.drbd2165.out‚0$ disk./.usageGB‚545.6311798095703 disk.rbd3.in‚0 disk.dm-0.out‚26995.37109375 disk.nvme0n1.in‚264.8125 disk.drbd2165.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚36.61776629207371% disk.nvme0c0n1.out‚26995.37109375 disk.rbd1.out‚96.41943359375 disk.nvme0c0n1.in‚264.8125 disk.dm-1.in‚0 disk.dm-1.out‚ 186.24609375 disk.dm-0.in‚264.8125`ÆSì“Ü‚(:ˆ šØçÒ“¼Û¤ gpu.0.gpu‚64.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚236.204' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.2408 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¬!©Ý‚(:ú šØçÒ±ÿ„§) disk./.usagePercent‚36.61777243625453% disk.nvme0n1.out‚27005.0849609375$ memory_percent‚95.47574520111084* proc.memory.percent‚17.142510414123535 network.recv‚ 1.392624e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚96.41943359375 disk.rbd3.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 265.52734375 disk.dm-1.in‚0 disk.dm-1.out‚ 186.41796875 disk.rbd1.in‚ 0.01171875 cpu‚26.005212785835518 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 network.sent‚ 9.786838e+06$ disk./.usageGB‚545.6312713623047' disk.nvme0c0n1.out‚27005.0849609375 disk.sda.out‚ 186.41796875 disk.sda3.out‚ 186.41796875( proc.memory.availableMB‚ 741.25390625 disk.drbd2165.in‚0 disk.dm-0.in‚ 265.52734375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 265.52734375# proc.memory.rssMB‚ 2808.62890625 disk.sda2.out‚0 disk.rbd3.out‚0" disk.dm-0.out‚27005.0849609375M[é2Þ- _wandbruntime‚5237  _runtime‚5237ùm5Éßà _step‚503 ' lr-AdamW/pg1‚0.00037923155639605006 ' lr-AdamW/pg2‚0.00037923155639605006 trainer/global_step‚6149 % _timestamp‚1.7842780457378886e+09 _runtime‚5237.863490599Ÿ§ÛÎàÈ ' lr-AdamW/pg1‚0.00037923155639605006 ' lr-AdamW/pg2‚0.00037923155639605006 trainer/global_step‚6149 % _timestamp‚1.7842780457378886e+09 _runtime‚5237.863490599 _step‚503÷nSg 2á- _wandbruntime‚5237  _runtime‚5237ùe×è'ââO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚929705847001088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48750652 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56959243 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105709895 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13247239 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941013791748096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205949525321728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105699916 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10775431 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205949506709504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6574422289920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56959243 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚941014150126592 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6502135812096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6574335810048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚929705864040448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7842780457389393e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚936198061409854 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚6149 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934511672975360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934511655936000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚941014150126592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚941013791748096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48750652 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13247497 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48750650 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212523947611648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508»qß„í'ãçO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24022667 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5237.864752602 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934511655936000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6492554958656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205949506709504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6574422289920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56959243 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚941014150126592 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6502135812096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6574335810048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚929705864040448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7842780457389393e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚936198061409854 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚6149 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934511672975360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934511655936000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚941014150126592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚941013791748096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48750652 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13247497 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48750650 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212523947611648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚929705847001088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48750652 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56959243 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105709895 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13247239 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941013791748096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205949525321728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105699916 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10775431 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24026394 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105716600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6503489785344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6503148439040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10777176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48758284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10777173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212558399507456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56968309 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚5238.475951059 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48758282 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚941162403790336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚941162403790336 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6575445427200 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56958318 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934658914004992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48758282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13249221 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212558288131072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6575352663040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚929852374712320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6493566012490 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934658914004992 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6503148439040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚929852357672960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚936345583135448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48758284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205982954080256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚505 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6503489785344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842780463498514e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105726593 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56958318 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24026645 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚6149Á»p~÷&æñM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6503489785344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6503148439040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10777176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48758284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10777173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212558399507456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56968309 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚5238.475951059 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48758282 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚941162403790336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚941162403790336 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6575445427200 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56958318 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934658914004992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48758282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13249221 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212558288131072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6575352663040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚929852374712320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6493566012490 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934658914004992 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6503148439040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚929852357672960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚936345583135448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48758284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205982954080256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚505 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6503489785344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842780463498514e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105726593 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56958318 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24026645 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚6149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105726593 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105716600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56968309 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24026394 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105716600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976ù6ÐuY2ç- _wandbruntime‚5238  _runtime‚5238)¥¶€èÿ /loss/FRt1/detection_loss‚0.5198734106021067 1loss/SRqqt1/detection_loss‚0.7750200919572233 _runtime‚5238.479942027 /loss/FRt1/assignment_loss‚1.673439201538815 0loss/FRt2/assignment_loss‚0.9925553410822618 1loss/SRqqt1/assignment_loss‚0.458319157100523 -loss/FRt2/detection_loss‚1.27293611396185 /loss/FBt1/detection_loss‚0.3672064804581014 /loss/FBt2/detection_loss‚0.9488140800967813 %loss/total_loss‚8.252275584260564 2loss/SRqqt2/assignment_loss‚0.5131693108379758 1loss/FBt1/assignment_loss‚0.04969583178523783 2loss/FBt2/assignment_loss‚0.009734339098658396 epoch‚15 trainer/global_step‚6149 _step‚506 1loss/SRqqt2/detection_loss‚0.6715122257410306 % _timestamp‚1.7842780463535304e+09 טd é„ 1loss/SRqqt2/detection_loss‚0.6715122257410306 % _timestamp‚1.7842780463535304e+09 /loss/FRt1/detection_loss‚0.5198734106021067 1loss/SRqqt1/detection_loss‚0.7750200919572233 _runtime‚5238.479942027 /loss/FRt1/assignment_loss‚1.673439201538815 0loss/FRt2/assignment_loss‚0.9925553410822618 1loss/SRqqt1/assignment_loss‚0.458319157100523 -loss/FRt2/detection_loss‚1.27293611396185 /loss/FBt1/detection_loss‚0.3672064804581014 /loss/FBt2/detection_loss‚0.9488140800967813 %loss/total_loss‚8.252275584260564 2loss/SRqqt2/assignment_loss‚0.5131693108379758 1loss/FBt1/assignment_loss‚0.04969583178523783 2loss/FBt2/assignment_loss‚0.009734339098658396 epoch‚15 trainer/global_step‚6149 _step‚506úôƒ¸Pê‚(:’ ©ØçÒý¥ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u"K ë‚(:ø ©ØçÒÇý¦§ disk.drbd2165.in‚0 network.recv‚ 1.396119e+06" disk.nvme0c0n1.in‚ 266.35546875( disk.nvme0c0n1.out‚27012.26123046875 disk.sda2.out‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚27012.26123046875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚36.61776910815659 disk.dm-1.out‚ 186.7734375 disk.rbd1.out‚96.48193359375# disk.dm-0.out‚27012.26123046875 disk.sda.out‚ 186.7734375 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚95.47438621520996 disk.sda.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2808.62890625 network.sent‚ 9.78976e+06$ disk./.usageGB‚545.6312217712402 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 266.35546875 cpu‚26.004392583142486 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 741.4765625 disk.sda3.out‚ 186.7734375 disk.nvme0n1.in‚ 266.35546875 disk.sda1.out‚0* proc.memory.percent‚17.142510414123535÷%H“ì‚(:ˆ ¸ØçÒˆ‡Û¤ gpu.0.gpu‚77.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.567' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤[^í‚(:ö ¸ØçÒ—ä¦ disk.sda.in‚0) disk./.usagePercent‚36.61776936416412 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 186.9453125 disk.nvme0n1.in‚ 266.35546875% disk.nvme0n1.out‚27018.1904296875 disk.sda.out‚ 186.9453125* proc.memory.percent‚17.142510414123535 disk.rbd1.out‚96.48193359375 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2808.62890625 network.sent‚ 9.854818e+06 network.recv‚ 1.400304e+06 disk.rbd1.in‚ 0.01171875 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 186.9453125 cpu‚26.003632436743306" disk.dm-0.out‚27018.1904296875' disk.nvme0c0n1.out‚27018.1904296875 disk.dm-0.in‚ 266.35546875 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚95.47333717346191' proc.memory.availableMB‚ 741.6484375 disk.rbd3.out‚0$ disk./.usageGB‚545.6312255859375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 266.355468754>í2î- _wandbruntime‚5277  _runtime‚5277ûùçwÉïà % _timestamp‚1.7842780854409132e+09 _runtime‚5277.566507681 _step‚507 ' lr-AdamW/pg1‚0.00037771838149982325 ' lr-AdamW/pg2‚0.00037771838149982325 trainer/global_step‚6199:cÅÎðÈ _runtime‚5277.566507681 _step‚507 ' lr-AdamW/pg1‚0.00037771838149982325 ' lr-AdamW/pg2‚0.00037771838149982325 trainer/global_step‚6199 % _timestamp‚1.7842780854409132e+09û¬NMÎ2ñ- _wandbruntime‚5277  _runtime‚5277ú#CÇòâO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13350659 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5277.567751169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚948426897110016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6553172653056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24213595 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49132252 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10862678 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214246371962880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207620963248128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6553172653056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49132250 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57413695 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214246483346432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842780854419565e+09 _step‚508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚948426538731520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚943574211490362 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49132252 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6625427326976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6552831313920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24213337 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49132250 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inaáÛ‡! ctive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106535968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚948426897110016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚937031380594688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6542830895674 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚943574569079612 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0²»sáí'óçO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6543171445564 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57403718 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5277.567751169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚948426897110016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6553172653056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24213595 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49132252 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10862678 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214246371962880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207620963248128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6553172653056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49132250 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57413695 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214246483346432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842780854419565e+09 _step‚508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚948426538731520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚943574211490362 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49132252 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6625427326976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6552831313920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24213337 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49132250 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106535968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚948426897110016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚937031380594688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6542830895674 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚943574569079612 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13350659 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207654392006656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106562645 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49139882 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10864426 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚942020965486592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106552652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚948575150773760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13352747 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚509 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49139882 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚948574792388096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49139884 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚943722090805318 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57422761 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚937177891266560 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942020965486592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6553843940864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942020948447232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106562645 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚948575150773760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214280831447552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚6199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49139884 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6553843940864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13353013 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚942020948447232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24217170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10864423 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6543841949396 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207654373394432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6554185287168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24217439 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57412770 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7842780860547698e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214280934435328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6626458053120 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214280934435328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6626458053120 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207654392006656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106562645 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49139882 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10864426 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚942020965486592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106552652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚948575150773760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13352747 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚509 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49139882 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚948574792388096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49139884 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚943722090805318 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57422761 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚937177891266560 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942020965486592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6553843940864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942020948447232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106562645 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚948575150773760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214280831447552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚6199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49139884 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6553843940864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13353013 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚942020948447232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24217170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10864423 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6543841949396 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207654373394432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6554185287168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24217439 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57412770 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7842780860547698e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49513850 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24404166 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949235758899200 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955839285714944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6593787932472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950950361570870 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚944356931227648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49513852 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚6249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215968885336064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57858170 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842781253917258e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57868147 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚955839285714944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123966464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950950719160120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10949928 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10949931 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49513850 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚955839644093440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6603868154880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24403908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13454235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚944356914188288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57858170 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955839644093440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚5317.520382947 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6603526815744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215969009302528 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105354240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚949235775938560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ÂXlXî'…èO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215968885336064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57858170 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842781253917258e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57868147 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚955839285714944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123966464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950950719160120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10949928 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10949931 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49513850 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚955839644093440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6603868154880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24403908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13454235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚944356914188288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57858170 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955839644093440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚5317.520382947 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6603526815744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215969009302528 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105354240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚949235775938560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49513852 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚949235775938560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107381999 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6676608128000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209292382562304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57868147 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13453980 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6593447382582 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107372020 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚949235758899200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6603868154880 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49513850 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24404166 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949235758899200 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955839285714944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6593787932472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950950361570870 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚944356931227648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49513852 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚6249€Åb…µ2†- _wandbruntime‚5318  _runtime‚5318vYF{H‡ìM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57867222 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49521482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955987897757184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6604880788992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49521484 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57877213 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24407929 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6594458436304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24407664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10951676 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951098240885826 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49521482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107388704 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57877213 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6677528924672 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚6249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6604539442688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6604539442688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209325829933056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6677621688832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚951097883296464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚955987539371520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107388704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49521484 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107398697 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107398697 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438º´'÷&ˆñM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57877213 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24407929 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6594458436304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24407664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10951676 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951098240885826 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49521482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107388704 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57877213 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6677528924672 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚6249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6604539442688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6604539442688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209325829933056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6677621688832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚951097883296464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚955987539371520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107388704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49521484 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107398697 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107398697 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57867222 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49521482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955987897757184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6604880788992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49521484 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 Î2‰- _wandbruntime‚5318  _runtime‚5318¯ÑpÊ Šƒ _runtime‚5318.132267514 2loss/FBt1/assignment_loss‚0.025001897332095936 0loss/FBt2/assignment_loss‚0.2160990257877897 trainer/global_step‚6249 2loss/SRqqt1/assignment_loss‚0.4501789926749595 1loss/SRqqt2/detection_loss‚0.6403755728954428 _step‚514 2loss/SRqqt2/assignment_loss‚0.5228149313018111 /loss/FRt1/detection_loss‚0.5681369873774385 0loss/FBt1/detection_loss‚0.43282452355940026 epoch‚15 0loss/FRt1/assignment_loss‚1.6659697692937454 0loss/FRt2/assignment_loss‚0.8717601060235458 /loss/FRt2/detection_loss‚1.2402617014482182 1loss/SRqqt1/detection_loss‚0.9915997955991431 /loss/FBt2/detection_loss‚1.0042842806886054 %loss/total_loss‚8.629307583982197 % _timestamp‚1.7842781260069237e+09î8!ª‹ˆ trainer/global_step‚6249 2loss/SRqqt1/assignment_loss‚0.4501789926749595 1loss/SRqqt2/detection_loss‚0.6403755728954428 _step‚514 2loss/SRqqt2/assignment_loss‚0.5228149313018111 /loss/FRt1/detection_loss‚0.5681369873774385 0loss/FBt1/detection_loss‚0.43282452355940026 epoch‚15 0loss/FRt1/assignment_loss‚1.6659697692937454 0loss/FRt2/assignment_loss‚0.8717601060235458 /loss/FRt2/detection_loss‚1.2402617014482182 1loss/SRqqt1/detection_loss‚0.9915997955991431 /loss/FBt2/detection_loss‚1.0042842806886054 %loss/total_loss‚8.629307583982197 % _timestamp‚1.7842781260069237e+09 _runtime‚5318.132267514 2loss/FBt1/assignment_loss‚0.025001897332095936 0loss/FBt2/assignment_loss‚0.2160990257877897‚<1¯Œ‚(:’ ôØçҠꤥ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÏÚ¡û‚(:ð ôØçÒŽœí©$ memory_percent‚95.47700881958008 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 269.44140625 cpu‚26.000395274339308 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚96.60693359375" disk.nvme0n1.out‚ 27051.9296875 disk.sda3.out‚ 187.98046875 disk.dm-1.out‚ 187.98046875 disk.rbd1.in‚ 0.01171875 disk.dm-0.out‚ 27051.9296875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 269.44140625& proc.memory.availableMB‚ 741.046875# proc.memory.rssMB‚ 2808.62890625* disk./.usagePercent‚36.617778580435356$ disk./.usageGB‚545.6313629150391 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 269.44140625* proc.memory.percent‚17.142510414123535 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 187.98046875 disk.rbd3.in‚0 network.sent‚ 9.931149e+06 network.recv‚ 1.415242e+06$ disk.nvme0c0n1.out‚ 27051.9296875 disk.dm-1.in‚0…¦†.’Ž‚(:‡ ƒÙçÒßÛÔ¤ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚288.775' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b©î‚(:ú ƒÙçÒ¸›§$ disk./.usageGB‚545.6314888000488 disk.sda1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 269.44140625 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 269.44140625 network.sent‚ 9.996146e+06 network.recv‚ 1.419971e+06 disk.nvme0n1.in‚ 269.44140625& disk.nvme0n1.out‚27055.29736328125 disk.sda.in‚0 disk.sda2.out‚0# disk.dm-0.out‚27055.29736328125 disk.sda1.out‚0 disk.sda.out‚ 188.1171875 disk.rbd1.out‚96.66943359375 disk.rbd3.in‚0* disk./.usagePercent‚36.617787028683985 disk.rbd0.out‚0( disk.nvme0c0n1.out‚27055.29736328125 disk.sda3.out‚ 188.1171875 disk.drbd2165.out‚0* proc.memory.percent‚17.142510414123535 cpu‚25.999632672123454 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd3.out‚0$ memory_percent‚95.47562599182129' proc.memory.availableMB‚ 741.2734375# proc.memory.rssMB‚ 2808.62890625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 188.1171875uT#òž‚(:“ ’ÙçÒ¯œã¤ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ‹Úò‘‚(:ç ’ÙçÒùÊø¦$ disk./.usageGB‚545.6290512084961 disk.rbd3.in‚0 disk.dm-0.in‚ 270.15625$ disk.nvme0n1.out‚27063.560546875 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2808.62890625! disk.dm-0.out‚27063.560546875 disk.nvme0n1.in‚ 270.15625 disk.sda.out‚ 188.359375 disk.sda3.out‚ 188.359375 disk.dm-1.out‚ 188.359375' proc.memory.availableMB‚ 741.2421875* proc.memory.percent‚17.142510414123535 network.sent‚ 9.99911e+06* disk./.usagePercent‚36.617623439869604 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 270.15625 disk.dm-1.in‚0 disk.rbd1.out‚96.66943359375 disk.rbd3.out‚0 disk.rbd0.out‚0 network.recv‚ 1.423186e+06 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 cpu‚25.998848540810293 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0$ memory_percent‚95.47581672668457& disk.nvme0c0n1.out‚27063.5605468758¤|ç2’- _wandbruntime‚5357  _runtime‚5357ÄWËîÆ“À & lr-AdamW/pg2‚0.0003746648869357452 trainer/global_step‚6299 $ _timestamp‚1.784278165280075e+09 _runtime‚5357.405667597 _step‚515 & lr-AdamW/pg1‚0.0003746648869357452‹Û—û˔Štrainer/global_step‚6299 $ _timestamp‚1.784278165280075e+09 _runtime‚5357.405667597 _step‚515 & lr-AdamW/pg1‚0.0003746648869357452 & lr-AdamW/pg2‚0.0003746648869357452ƒÃeý2•- _wandbruntime‚5357  _runtime‚5357y=–]è'–âO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108208072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963252032698368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217691529866240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚963252391076864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚958326869240628 _step‚516 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11037178 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚956597810380800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚951682447781888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49895450 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6727690765312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6654563656704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6727600091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚6299 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚963252032698368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚210963820488704 _runtime‚5357.406909522 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚956597827420160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49895452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚958326511651378 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49895452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654222317568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58312622 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13556723 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚951682464821248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11037178 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚956597810380800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚951682447781888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49895—ïeê450 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6727690765312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6654563656704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6727600091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚6299 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚963252032698368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚210963820488704 _runtime‚5357.406909522 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚956597827420160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49895452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚958326511651378 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49895452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654222317568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58312622 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13556723 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚951682464821248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963252032698368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217691529866240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚963252391076864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚958326869240628 _step‚516 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24597480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108234749 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚963400644740608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6655576290816 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚951828975493120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217725975355904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655234944512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚956745051410432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58331665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6728614732288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚963400286354944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108234749 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49903084 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚958474033376972 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚6299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13558557 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210997249247232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58331665 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7842781658922698e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6645074923212 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49903082 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24597735 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚951828958453760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217725863979520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6655576290816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49903084 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24597480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108234749 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚963400644740608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6655576290816 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚951828975493120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217725975355904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655234944512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚956745051410432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58331665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6728614732288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚963400286354944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108234749 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49903084 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚958474033376972 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚6299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13558557 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210997249247232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58331665 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7842781658922698e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6645074923212 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49903082 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24597735 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚951828958453760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002……‹ã%2›- _wandbruntime‚5358  _runtime‚5358Íûz‡ œ… 0loss/FRt2/assignment_loss‚0.8820543737019111 3loss/FBt1/assignment_loss‚0.0003109985074916918 /loss/FRt2/detection_loss‚1.1560758220174305 1loss/SRqqt2/detection_loss‚0.6182808048637214 /loss/FBt2/detection_loss‚1.5968972842674702 %loss/total_loss‚8.988001946812362 trainer/global_step‚6299 2loss/SRqqt2/assignment_loss‚0.5341342837279308 epoch‚15 % _timestamp‚1.7842781658949747e+09 _runtime‚5358.020311767 _step‚518 2loss/SRqqt1/assignment_loss‚0.4800622347442927 1loss/SRqqt1/detection_loss‚0.9725355360337913 0loss/FRt1/assignment_loss‚1.7347928015850362 2loss/FBt2/assignment_loss‚0.001975688909346395 /loss/FRt1/detection_loss‚0.5395992215685912 /loss/FBt1/detection_loss‚0.4712828968853488ÊgUŠ trainer/global_step‚6299 2loss/SRqqt2/assignment_loss‚0.5341342837279308 epoch‚15 % _timestamp‚1.7842781658949747e+09 _runtime‚5358.020311767 _step‚518 2loss/SRqqt1/assignment_loss‚0.4800622347442927 1loss/SRqqt1/detection_loss‚0.9725355360337913 0loss/FRt1/assignment_loss‚1.7347928015850362 2loss/FBt2/assignment_loss‚0.001975688909346395 /loss/FRt1/detection_loss‚0.5395992215685912 /loss/FBt1/detection_loss‚0.4712828968853488 0loss/FRt2/assignment_loss‚0.8820543737019111 3loss/FBt1/assignment_loss‚0.0003109985074916918 /loss/FRt2/detection_loss‚1.1560758220174305 1loss/SRqqt2/detection_loss‚0.6182808048637214 /loss/FBt2/detection_loss‚1.5968972842674702 %loss/total_loss‚8.988001946812362†ÔcJ“ž‚(:ˆ ¡ÙçÒü£Ž£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kw·¹Ÿ‚(:÷ ¡ÙçÒú½¡§ disk.rbd1.out‚96.79443359375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0$ memory_percent‚95.47581672668457* disk./.usagePercent‚36.617861782883985* proc.memory.percent‚17.142510414123535 disk.rbd3.out‚0 disk.sda.out‚ 188.64453125 disk.sda3.in‚0' proc.memory.availableMB‚ 741.2421875 cpu‚25.99806362780806 disk.sda2.out‚0# proc.memory.rssMB‚ 2808.62890625 network.recv‚ 1.426788e+06 disk.rbd3.in‚0 disk.dm-0.in‚ 270.984375 disk.nvme0n1.in‚ 270.984375& disk.nvme0n1.out‚27070.87841796875$ disk./.usageGB‚545.6326026916504 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 188.64453125# disk.dm-0.out‚27070.87841796875 disk.sda.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 188.64453125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 270.984375( disk.nvme0c0n1.out‚27070.87841796875 network.sent‚ 1.0017157e+07¨’)v‘ ‚(:† °ÙçÒ¢›â¤ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ä ,¡‚(:û °ÙçÒ´Ù‹§ network.sent‚ 1.0069223e+07* disk./.usagePercent‚36.617909400285356$ disk./.usageGB‚545.6333122253418 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 271.69921875# proc.memory.rssMB‚ 2808.62890625 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚27079.5986328125 disk.sda.out‚ 189.08984375 disk.drbd2165.in‚0" disk.dm-0.out‚27079.5986328125' proc.memory.availableMB‚ 741.3046875 cpu‚25.999129603110404% disk.nvme0n1.out‚27079.5986328125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 271.69921875 disk.sda3.in‚0 disk.sda3.out‚ 189.08984375 disk.dm-1.out‚ 189.08984375$ memory_percent‚95.47543525695801 network.recv‚ 1.431359e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚96.79443359375 disk.dm-0.in‚ 271.69921875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875* proc.memory.percent‚17.142510414123535q}æ‹.¢- _wandbruntime‚5397  _runtime‚2*Cd5397E–u‚Æ£À & lr-AdamW/pg1‚0.0003731247633881439 & lr-AdamW/pg2‚0.0003731247633881439 trainer/global_step‚6349 $ _timestamp‚1.784278205371879e+09 _runtime‚5397.497386804 _step‚519JÛýoˤŠ_runtime‚5397.497386804 _step‚519 & lr-AdamW/pg1‚0.0003731247633881439 & lr-AdamW/pg2‚0.0003731247633881439 trainer/global_step‚6349 $ _timestamp‚1.784278205371879e+09‡ÄE2¥- _wandbruntime‚5397  _runtime‚5397Sàñ‚è'¦âO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13658676 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚965702661731886 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚959007998414848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6704917819392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109054103 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7842782053729503e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212635277027328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109044124 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50277050 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6778909878784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6695020906288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6705259158528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970665138060288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚963959861862400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚5397.498699322 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58777051 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219414186906112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚970665138060288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212635258415104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24783377 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219414079716864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970664779681792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50277050 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58767074 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6778909878784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6695020906288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6705259158528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚970665138060288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚963959861862400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚5397.498699322 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58777051 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219414186906112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚970665138060288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212635258415104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24783377 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219414079716864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970664779681792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50277050 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58767074 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚970813391724032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚964107119931392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58776126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚959154509086720 _step‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50284682 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24787127 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109070801 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109060808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6705930446336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109070801 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212668687173632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6705930446336 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24786866 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58786117 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6779834238976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58776126 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7842782059855876e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚959154492047360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚5398.111419101 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50284684 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚970813391724032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13660693 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219448521412608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚964107102892032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6706271792640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11126176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0qL©ªñM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7842782059855876e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚959154492047360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚5398.111419101 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50284684 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚970813391724032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13660693 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219448521412608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚964107102892032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6706271792640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11126176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚970813391724032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚964107119931392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58776126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚959154509086720 _step‚521 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50284682 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24787127 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109070801 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109060808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6705930446336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109070801 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212668687173632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6705930446336 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24786866 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58786117 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6779834238976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58776126 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169‰‚«ìÌ2«- _wandbruntime‚5398  _runtime‚5398]I`™¬ˆ 3loss/SRqqt1/assignment_loss‚0.45863153565818826 /loss/FRt2/detection_loss‚1.2333305941329176 1loss/SRqqt1/detection_loss‚0.9111056856299732 1loss/SRqqt2/detection_loss‚0.6614445303381823 %loss/total_loss‚8.239984395724752 epoch‚15 % _timestamp‚1.7842782059886105e+09 _step‚522 _runtime‚5398.113933708 0loss/FRt1/assignment_loss‚1.6697904854016268 2loss/FBt1/assignment_loss‚0.028350422691627568 3loss/FBt2/assignment_loss‚0.0003766577538423614 /loss/FRt1/detection_loss‚0.5201267166752352 0loss/FBt1/detection_loss‚0.49870826054919654 trainer/global_step‚6349 0loss/FRt2/assignment_loss‚0.9080641442903112 3loss/SRqqt2/assignment_loss‚0.49691429249292796 /loss/FBt2/detection_loss‚0.85314107011072346 ¨­ 3loss/SRqqt1/assignment_loss‚0.45863153565818826 /loss/FRt2/detection_loss‚1.2333305941329176 1loss/SRqqt1/detection_loss‚0.9111056856299732 1loss/SRqqt2/detection_loss‚0.6614445303381823 %loss/total_loss‚8.239984395724752 epoch‚15 % _timestamp‚1.7842782059886105e+09 _step‚522 _runtime‚5398.113933708 0loss/FRt1/assignment_loss‚1.6697904854016268 2loss/FBt1/assignment_loss‚0.028350422691627568 3loss/FBt2/assignment_loss‚0.0003766577538423614 /loss/FRt1/detection_loss‚0.5201267166752352 0loss/FBt1/detection_loss‚0.49870826054919654 trainer/global_step‚6349 0loss/FRt2/assignment_loss‚0.9080641442903112 3loss/SRqqt2/assignment_loss‚0.49691429249292796 /loss/FBt2/detection_loss‚0.8531410701107234ŠUü)ž®‚(:“ ¿ÙçÒɉѤ gpu.0.gpu‚90.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BA¢Àø¯‚(:í ¿ÙçÒÆÌͦ proc.cpu.threads‚17$ memory_percent‚96.97952270507812# disk.dm-0.out‚27086.32080078125 disk.nvme0n1.in‚ 272.52734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚27086.32080078125 disk.sda2.out‚0& disk.nvme0n1.out‚27086.32080078125 disk.sda1.in‚0 disk.rbd1.out‚96.85693359375 disk.rbd0.out‚0 disk.sda1.out‚0 network.recv‚ 1.434592e+06 disk.rbd3.in‚0# proc.memory.availableMB‚494.875 cpu‚26.000054962620474 disk.sda.out‚189.3125 disk.dm-1.in‚0$ disk./.usageGB‚545.6323471069336" disk.nvme0c0n1.in‚ 272.52734375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚189.3125 disk.sda2.in‚0 network.sent‚ 1.0085228e+07) disk./.usagePercent‚36.61784463037919 disk.rbd1.in‚ 0.01171875* proc.memory.percent‚17.142510414123535 disk.dm-0.in‚ 272.52734375 disk.dm-1.out‚189.3125 disk.rbd3.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2808.62890625óñf“°‚(:ˆ ÎÙçÒÊËÔ¤ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä2˜Mÿ±‚(:ô ÎÙçÒà´û¦ disk.sda1.in‚0 disk.sda.out‚ 189.53125 disk.nvme0n1.in‚ 272.52734375 disk.sda.in‚0 disk.dm-1.out‚ 189.53125 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 272.52734375 proc.cpu.threads‚17 disk.sda3.out‚ 189.53125 disk.dm-1.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.142510414123535 network.sent‚ 1.0137032e+07 disk.sda1.out‚0( disk.nvme0c0n1.out‚27092.26904296875 disk.rbd1.out‚96.85693359375( proc.memory.availableMB‚ 495.18359375$ disk./.usageGB‚545.6261863708496& disk.nvme0n1.out‚27092.26904296875 disk.drbd2165.out‚0$ memory_percent‚96.97763919830322 cpu‚26.00101274502965 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚27092.26904296875# proc.memory.rssMB‚ 2808.62890625 network.recv‚ 1.438119e+06" disk.nvme0c0n1.in‚ 272.52734375 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚36.61743117821139]Ùâs²‚(:„ ÝÙçÒöÅϤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²²†Nÿ³‚(:ô ÝÙçÒ¡ÄÛ¦ disk.rbd1.in‚ 0.01171875' disk.nvme0c0n1.out‚27102.4013671875* proc.memory.percent‚17.142510414123535 disk.sda2.out‚0( disk./.usagePercent‚36.6180368920374 disk.rbd3.out‚0 disk.rbd1.out‚96.91943359375 disk.dm-0.in‚ 273.2421875% disk.nvme0n1.out‚27102.4013671875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2808.62890625 disk.dm-1.in‚0 disk.dm-1.out‚ 189.7265625! disk.nvme0c0n1.in‚ 273.2421875 disk.sda3.out‚ 189.7265625( proc.memory.availableMB‚ 495.17578125 network.recv‚ 1.441616e+06 disk.rbd3.in‚0 disk.nvme0n1.in‚ 273.2421875 disk.sda.in‚0 disk.sda.out‚ 189.7265625 disk.sda3.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚545.6352119445801" disk.dm-0.out‚27102.4013671875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚96.97768688201904 network.sent‚ 1.0141289e+07 disk.sda1.out‚0 disk.drbd2165.in‚0 cpu‚26.002153917479866^‘´2´- _wandbruntime‚5438  _runtime‚5438Ûµ .ȵ ' lr-AdamW/pg1‚0.00037157585403229466 ' lr-AdamW/pg2‚0.00037157585403229466 trainer/global_step‚6399 % _timestamp‚1.7842782458868735e+09 _runtime‚ 5438.01245885 _step‚523îηžÍ¶Ç _runtime‚ 5438.01245885 _step‚523 ' lr-AdamW/pg1‚0.00037157585403229466 ' lr-AdamW/pg2‚0.00037157585403229466 trainer/global_step‚6399 % _timestamp‚1.7842782458868735e+09‹*dþ2·- _wandbruntime‚5438  _runtime‚5438¨þôìç'¸áO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11211681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11211678 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚971321930383360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6755613321216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109880176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50658650 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13761084 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7842782458879251e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50658652 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚524 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6755613321216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚971321930383360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚978077526665216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6830046177280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6745296843306 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚971321913344000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59221526 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚973078811812394 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978077526665216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109890155 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24973040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚966333532008448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59231503 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6745637393196 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221136761131008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214306714953728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13761359 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚978077885043712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚ 5438.01371408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50658652 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6755954660352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚973079169401644 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983¹– ±¹æO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109880176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50658650 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13761084 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7842782458879251e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50658652 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _step‚524 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6755613321216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚971321930383360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚978077526665216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6830046177280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6745296843306 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚971321913344000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59221526 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚973078811812394 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978077526665216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109890155 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24973040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚966333532008448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59231503 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6745637393196 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221136761131008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214306714953728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13761359 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚978077885043712 G?DeviceStatsMonitor.on_train_batch_start/inactive_splitŸJ ; .all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚ 5438.01371408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50658652 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6755954660352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚973079169401644 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11211681 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11211678 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚971321930383360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6755613321216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Œ¨D€2º- _wandbruntime‚5438  _runtime‚5438†ÜÞ¼ò&»ìM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109906853 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚971469171412992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109896860 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221171212264960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚971469154373632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109906853 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109896860 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50666282 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214340143712256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6756625948160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚966480042680320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50666284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚971469171412992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚973226333537988 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚5438.627556953 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚978226138707456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50666282 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59230578 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11213426 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6746307897028 _step‚525 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚978226138707456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6830969497088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24976501 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11213423 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13763078 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚966480025640960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚971469154373632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6831068552704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6399 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6756625948160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24976765 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221171094597120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59240569 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13763339 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0-™÷&¼ñM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24976501 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11213423 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13763078 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚966480025640960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚971469154373632 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6831068552704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6399 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6756625948160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24976765 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221171094597120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59240569 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13763339 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109906853 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚971469171412992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109896860 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221171212264960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚971469154373632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109906853 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109896860 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50666282 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214340143712256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6756625948160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚966480042680320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50666284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚971469171412992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚973226333537988 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚5438.627556953 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚978226138707456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50666282 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59230578 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11213426 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6746307897028 _step‚525 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚978226138707456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6830969497088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 P&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08232954889535904 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.32343900203704834 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7237583994865417 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 .particle/accuracy_3_of_4‚0.994953453540802 /particle/accuracy_4_of_4‚0.9337437152862549 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948812127113342 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02982163429260254 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6774299740791321 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6856953501701355 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6961712837219238 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7913427948951721 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 jet/accuracy_2_of_5‚1 +particle/sensitivity‚0.6124483942985535 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9878763556480408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5458332896232605 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3239220082759857 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.055398136377334595 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7133545875549316 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3655652403831482 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029592877253890038 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7000823616981506 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7714663743972778 *jet/accuracy_1_of_1‚0.3691168427467346 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 % _timestamp‚1.7842782644344456e+09 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3138236403465271 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7015020251274109 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_5_of_5‚0 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.042636338621377945 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13166667520999908 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.048508938401937485 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975960850715637 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5107812881469727 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7517384886741638 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7330039143562317 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2465859204530716 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7551056742668152 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,jet/accuracy_4_of_4‚0.013899499550461769 +particle/specificity‚0.9089317321777344 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.28125 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7322627902030945 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5599169135093689 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17158085107803345 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6798189282417297 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15857471525669098 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8007935881614685 *jet/accuracy_1_of_2‚0.8174569010734558 jet/accuracy_1_of_5‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30508360266685486 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26732736825942993 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3159841001033783 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027356170117855072 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43098708987236023 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.37204140424728394 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26366671919822693 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8580063581466675 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8872750401496887 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5066609382629395 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5008770227432251 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40075182914733887 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.692706823348999 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 _step‚527 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Pu^¿Ô†!rity/0FRt2SRqqt0FBt/event_purity‚0.5255834460258484 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7451900839805603 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 7validation_average_jet_accuracy‚0.40526920557022095 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5653640031814575 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7902477979660034 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6717513203620911 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7081459760665894 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6653928160667419 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4000341296195984 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7522577047348022 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3251085877418518 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3406818211078644 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6104311347007751 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26738911867141724 particle/accuracy_2_of_3‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970593452453613 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44109511375427246 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_2_of_3‚0.5843373537063599 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949179291725159 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08177758753299713 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7748299837112427 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23416689038276672 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2492285966873169 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03114178404211998 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9710875749588013 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚6399 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7163434624671936 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9944109320640564 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25189390778541565 +jet/accuracy_3_of_3‚0.06356582045555115 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.305904358625412 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893204569816589 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903962016105652 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.732254147529602 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7874528169631958 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873440265655518 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7560683488845825 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9956257939338684 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25780951976776123 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9182080030441284 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956494569778442 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23443494737148285 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03114178404211998 (particle/accuracy‚0.8452613353729248 jet/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7625983357429504 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.664213240146637 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚15 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5622660517692566 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8554292321205139 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963951110839844 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_3‚0.9361522197723389 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969664216041565 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8632798790931702 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9247472882270813 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.274161696434021 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6770616769790649 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7165000438690186 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5719584226608276 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04638579115271568 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3159841001033783 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.989403486251831 _runtime‚5456.560647344 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09372974187135696 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6717513203620911 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7030572891235352 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7433772683143616 'particle/f_score‚0.6296049356460571 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23936355113983154 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7892497777938843 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18743671476840973 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1740691214799881 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6564706563949585 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7836047410964966 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22983603179454803 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6484683156013489 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_5‚1 ±8ÂÑ=ÄË{ 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6770616769790649 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7165000438690186 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5719584226608276 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04638579115271568 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3159841001033783 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.989403486251831 _runtime‚5456.560647344 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09372974187135696 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6717513203620911 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7030572891235352 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7433772683143616 'particle/f_score‚0.6296049356460571 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23936355113983154 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7892497777938843 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18743671476840973 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1740691214799881 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6564706563949585 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7836047410964966 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22983603179454803 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6484683156013489 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4236111044883728 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2773459255695343 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02760879509150982 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28902682662010193 +jet/accuracy_2_of_2‚0.16264952719211578 )loss/val_total_loss‚8.338372412111655 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6704466938972473 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8554292321205139 *jet/accuracy_2_of_4‚0.7945811152458191 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9921453595161438 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6168674230575562 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08232954889535904 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.32343900203704834 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7237583994865417 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 .particle/accuracy_3_of_4‚0.994953453540802 /particle/accuracy_4_of_4‚0.9337437152862549 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948812127113342 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02982163429260254 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6774299740791321 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6856953501701355 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6961712837219238 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7913427948951721 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 jet/accuracy_2_of_5‚1 +particle/sensitivity‚0.6124483942985535 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9878763556480408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5458332896232605 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3239220082759857 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.055398136377334595 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7133545875549316 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3655652403831482 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029592877253890038 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7000823616981506 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7714663743972778 *jet/accuracy_1_of_1‚0.3691168427467346 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 % _timestamp‚1.7842782644344456e+09 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3138236403465271 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7015020251274109 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_5_of_5‚0 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.042636338621377945 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13166667520999908 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.048508938401937485 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975960850715637 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5107812881469727 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7517384886741638 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7330039143562317 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2465859204530716 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7551056742668152 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,jet/accuracy_4_of_4‚0.013899499550461769 +particle/specificity‚0.9089317321777344 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.28125 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7322627902030945 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5599169135093689 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17158085107803345 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6798189282417297 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15857471525669098 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8007935881614685 *jet/accuracy_1_of_2‚0.8174569010734558 jet/accuracy_1_of_5‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30508360266685486 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26732736825942993 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3159841001033783 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027356170117855072 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43098708987236023 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.37204140424728394 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26366671919822693 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8580063581466675 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8872750401496887 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5066609382629395 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5008770227432251 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40075182914733887 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.692706823348999 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 _step‚527 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5255834460258484 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7451900839805603 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 7validation_average_jet_accuracy‚0.40526920557022095 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5653640031814575 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7902477979660034 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6717513203620911 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7081459760665894 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6653928160667419 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4000341296195984 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7522577047348022 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3251085877418518 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3406818211078644 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6104311347007751 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26738911867141724 particle/accuracy_2_of_3‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970593452453613 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44109511375427246 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_2_of_3‚0.5843373537063599 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949179291725159 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08177758753299713 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7748299837112427 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23416689038276672 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2492285966873169 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03114178404211998 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9710875749588013 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚6399 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7163434624671936 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9944109320640564 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25189390778541565 +jet/accuracy_3_of_3‚0.06356582045555115 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.305904358625412 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9893204569816589 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903962016105652 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.732254147529602 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7874528169631958 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873440265655518 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7560683488845825 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9956257939338684 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25780951976776123 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9182080030441284 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956494569778442 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23443494737148285 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03114178404211998 (particle/accuracy‚0.8452613353729248 jet/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7625983357429504 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.664213240146637 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚15 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5622660517692566 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8554292321205139 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963951110839844 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_3‚0.9361522197723389 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969664216041565 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8632798790931702 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9247472882270813 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.274161696434021 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125s®y˜œÅ‚(:‘ ûÙçÒ“‚°¥ gpu.0.gpu‚35.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚292.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.54859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü›÷Æ‚(:ì ûÙçÒôù⧠network.sent‚ 1.022011e+07 disk.sda3.out‚ 190.28125$ memory_percent‚95.47038078308105' proc.memory.availableMB‚ 742.1328125 disk.nvme0n1.in‚ 274.78515625& disk.nvme0c0n1.out‚27366.287109375 disk.sda2.in‚0 disk.dm-0.in‚ 274.78515625! disk.dm-0.out‚27366.287109375 cpu‚26.0142104003297 proc.cpu.threads‚17 network.recv‚ 1.449511e+06$ disk.nvme0n1.out‚27366.287109375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 190.28125 disk.rbd1.out‚96.98193359375$ disk./.usageGB‚545.6369132995605 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 190.28125* disk./.usagePercent‚36.618151071397676" disk.nvme0c0n1.in‚ 274.78515625 disk.rbd3.out‚0# proc.memory.rssMB‚ 2817.87109375 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.198920249938965 disk.sda1.in‚0 disk.sda2.out‚0`LÙ¨“Ç‚(:ˆ ŠÚçÒÁÿæ¤ gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²;ê5È‚(:õ ŠÚçÒÙÍŒ§ disk.sda3.out‚ 190.44140625 disk.dm-1.out‚ 190.44140625 disk.rbd1.out‚96.98193359375# proc.memory.rssMB‚ 2817.87109375 disk.rbd0.out‚0 disk.sda.out‚ 190.44140625 network.sent‚ 1.0264626e+07 disk.sda2.in‚0! disk.dm-0.out‚27376.607421875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 275.61328125& disk.nvme0c0n1.out‚27376.607421875* proc.memory.percent‚17.198920249938965 disk.rbd0.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚27376.607421875 network.recv‚ 1.453703e+06 disk.rbd1.in‚ 0.01171875# memory_percent‚95.4688549041748' proc.memory.availableMB‚ 742.3828125) disk./.usagePercent‚36.61815388748055 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 275.61328125$ disk./.usageGB‚545.6369552612305 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 275.61328125 cpu‚26.01490299965775 proc.cpu.threads‚17 disk.sda1.out‚09ÉQ†“É‚(:ˆ ™ÚçÒùçý£ gpu.0.gpu‚69.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Å8ÑÊ‚(:ö ™ÚçÒø‰þ¥ disk.drbd2165.in‚0 network.recv‚ 1.457575e+06 disk.sda.out‚ 190.62109375 disk.rbd3.in‚0$ memory_percent‚95.47157287597656# proc.memory.rssMB‚ 2817.87109375 proc.cpu.threads‚17 disk.sda3.out‚ 190.62109375 disk.dm-1.in‚0 disk.dm-1.out‚ 190.62109375 network.sent‚ 1.0271856e+07 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚97.04443359375$ disk./.usageGB‚545.6358451843262 disk.nvme0n1.in‚ 275.61328125 disk.dm-0.in‚ 275.61328125 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0" disk.dm-0.out‚27625.2958984375 disk.rbd1.in‚ 0.01171875" disk.nvme0c0n1.in‚ 275.61328125' disk.nvme0c0n1.out‚27625.2958984375) disk./.usagePercent‚36.61807938928809% disk.nvme0n1.out‚27625.2958984375 disk.sda1.in‚0* proc.memory.percent‚17.198920249938965 cpu‚26.01595745687072 disk.rbd0.in‚0 disk.rbd3.out‚0$ proc.memory.availableMB‚741.9375ËFI2Ë- _wandbruntime‚5498  _runtime‚5498Ø”§²ÉÌà % _timestamp‚1.7842783063896747e+09 _runtime‚5498.515306885 _step‚528 ' lr-AdamW/pg1‚0.00037001825835175097 ' lr-AdamW/pg2‚0.00037001825835175097 trainer/global_step‚6449ûQ†äÎÍÈ ' lr-AdamW/pg1‚0.00037001825835175097 ' lr-AdamW/pg2‚0.00037001825835175097 trainer/global_step‚6449 % _timestamp‚1.7842783063896747e+09 _runtime‚5498.515306885 _step‚528Wß³–2Î- _wandbruntime‚5498  _runtime‚5498çþ 6™ ÏáO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59746305 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215994046367744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6824383370240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6824042031104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚5498.519403037 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚987437573507072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚982381344832216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.aljCDNlocated‚980613548515328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6899360422912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11301451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚6449 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚980613531475968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚987437931885568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6813973605338 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59746305 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215994046367744 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6824383370240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6824042031104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚5498.519403037 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚987437573507072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚982381344832216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚980613548515328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6899360422912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11301451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚6449 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚980613531475968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚987437931885568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6813973605338 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6900394084864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59765348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11303196 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚222927786932736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6814984659172 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚982529224147172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51168548 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59755357 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110933898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25202043 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51168548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842783070025969e+09îûÓñM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚222927887823360 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚222927786932736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6814984659172 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚982529224147172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51168548 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59755357 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110933898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25202043 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51168548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842783070025969e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51168550 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚980760789544960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6900394084864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59765348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11303196 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60210734 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚994850678868992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚989757852501974 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6864249542356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994850320490496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚987975582957568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚987975599996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217665484294144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚987975599996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14001393 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994850678868992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11388701 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224616044587008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚982893245370368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11388698 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60210734 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111753252 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25390094 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51542518 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60200757 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217665502906368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6950560292864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚987975582957568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224616147581952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6499 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25389823 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842783468808503e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111743273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6875078872064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6864590092246 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60200757 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51542518 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14001125 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6874737532928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51542516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6950644675584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60210734 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚994850678868992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚989757852501974 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6864249542356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994850320490496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚987975582957568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚987975599996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217665484294144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚987975599996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14001393 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994850678868992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11388701 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224616044587008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚982893245370368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11388698 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60210734 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111753252 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25390094 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51542518 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60200757 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217665502906368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6950560292864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚987975582957568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224616147581952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6499 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25389823 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842783468808503e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111743273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6875078872064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6864590092246 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60200757 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51542518 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14001125 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6874737532928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51542516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6950644675584 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51550148 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6951587072512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25393624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111759957 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.currò¤c5â ent‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25393871 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6951669350912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚988122841026560 % _timestamp‚1.7842783474933727e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6876091506176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11390446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224650500125184 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60209809 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚988122823987200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111769950 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60219800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚983039773081600 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60219800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60209809 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6865601146080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6865260596078 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚983039756042240 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111769950 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14003425 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111759957 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0džEÚ÷&åñM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚983039756042240 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111769950 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14003425 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111759957 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚994998932532736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚988122823987200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51550148 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6951587072512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25393624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111759957 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25393871 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6951669350912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚988122841026560 % _timestamp‚1.7842783474933727e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6876091506176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11390446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224650500125184 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60209809 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚988122823987200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111769950 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60219800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚983039773081600 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60219800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60209809 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6865601146080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6865260596078–Té â2æ- _wandbruntime‚5539  _runtime‚5539ôT¾Ôç‚ 3loss/SRqqt1/assignment_loss‚0.47644181447155465 2loss/FBt1/assignment_loss‚0.016933780877714303 1loss/SRqqt1/detection_loss‚0.9056476005862056 /loss/FBt2/detection_loss‚0.7752165080455597 trainer/global_step‚6499 _step‚535 0loss/FRt2/assignment_loss‚0.8979804577146258 2loss/SRqqt2/assignment_loss‚0.5518112791895832 1loss/SRqqt2/detection_loss‚0.6985315486107219 /loss/FBt1/detection_loss‚0.5878596222147101 %loss/total_loss‚8.542332986738526 epoch‚16 1loss/FBt2/assignment_loss‚0.01633054526610067 .loss/FRt1/detection_loss‚0.569885899116849 /loss/FRt2/detection_loss‚1.3363606668074084 % _timestamp‚1.7842783474968655e+09 _runtime‚ 5539.62218854 0loss/FRt1/assignment_loss‚1.7093332638374914Ïjxå è‡ 3loss/SRqqt1/assignment_loss‚0.47644181447155465 2loss/FBt1/assignment_loss‚0.016933780877714303 1loss/SRqqt1/detection_loss‚0.9056476005862056 /loss/FBt2/detection_loss‚0.7752165080455597 trainer/global_step‚6499 _step‚535 0loss/FRt2/assignment_loss‚0.8979804577146258 2loss/SRqqt2/assignment_loss‚0.5518112791895832 1loss/SRqqt2/detection_loss‚0.6985315486107219 /loss/FBt1/detection_loss‚0.5878596222147101 %loss/total_loss‚8.542332986738526 epoch‚16 1loss/FBt2/assignment_loss‚0.01633054526610067 .loss/FRt1/detection_loss‚0.569885899116849 /loss/FRt2/detection_loss‚1.3363606668074084 % _timestamp‚1.7842783474968655e+09 _runtime‚ 5539.62218854 0loss/FRt1/assignment_loss‚1.7093332638374914—³ÍÉžé‚(:“ ÕÚçÒè—»¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bãžöê‚(:ë ÕÚçÒ½çÚ§ network.sent‚ 1.0347986e+07 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 277.87109375 cpu‚26.019477829310166 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚95.46952247619629 disk.sda3.out‚ 191.65625 disk.rbd1.out‚97.16943359375 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 191.65625# disk.nvme0n1.out‚27651.23046875 disk.rbd0.out‚0% disk.nvme0c0n1.out‚27651.23046875' proc.memory.availableMB‚ 742.2734375 disk.rbd3.in‚0* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 network.recv‚ 1.471976e+06 disk.dm-1.in‚0 disk.dm-0.in‚ 277.87109375 disk.dm-0.out‚27651.23046875) disk./.usagePercent‚36.61808706951412$ disk./.usageGB‚545.6359596252441 disk.nvme0n1.in‚ 277.87109375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 191.65625# proc.memory.rssMB‚ 2817.87109375 disk.sda.in‚0&Š·Šœë‚(:‘ äÚçÒªÌÓ¢ gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«cé£ì‚(:ý äÚçÒä·§ network.sent‚ 1.0410937e+07 disk.nvme0n1.in‚ 278.69921875& disk.nvme0n1.out‚27660.73291015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚97.16943359375 disk.rbd3.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 191.88671875* proc.memory.percent‚17.198920249938965 disk.dm-1.in‚0 disk.dm-0.in‚ 278.69921875# proc.memory.rssMB‚ 2817.87109375 proc.cpu.threads‚17) disk./.usagePercent‚36.61808297339357$ disk./.usageGB‚545.6358985900879 network.recv‚ 1.476356e+06 disk.sda1.in‚0 disk.sda.out‚ 191.88671875 disk.sda3.in‚0# disk.dm-0.out‚27660.73291015625 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 278.69921875( disk.nvme0c0n1.out‚27660.73291015625 disk.sda2.in‚0 disk.sda3.out‚ 191.88671875 disk.rbd3.out‚0$ memory_percent‚95.46952247619629' proc.memory.availableMB‚ 742.2734375 cpu‚26.0204562512078222â2í- _wandbruntime‚5579  _runtime‚5579XmxpÉîà _runtime‚5579.252331232 _step‚536 ' lr-AdamW/pg1‚0.00036687740873390105 ' lr-AdamW/pg2‚0.00036687740873390105 trainer/global_step‚6549 % _timestamp‚1.7842783871267443e+09 HR’ÎïÈ ' lr-AdamW/pg1‚0.00036687740873390105 ' lr-AdamW/pg2‚0.00036687740873390105 trainer/global_step‚6549 % _timestamp‚1.7842783871267443e+09 _runtime‚5579.252331232 _step‚536˜~™2ð- _wandbruntime‚5579  _runtime‚5579àfšK$ñæO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25581159 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚219336940832768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226338886765568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚537 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1002263067473920 trainer/global_step‚6549 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚995337651478528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚995337634439168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚997134002582482 $ _timestamp‚1.784278387127785e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚990218778963968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11475948 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25581433 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14105482 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14105211 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6925774373888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51924118 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1002263067473920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60665186 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226339006537728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚997133644993232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚995337651478528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6925774373888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508ñDñ'òëO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25581433 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14105482 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14105211 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6925774373888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51924118 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1002263067473920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60665186 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226339006537728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚997133644993232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚995337651478528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6925774373888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25581159 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚219336940832768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226338886765568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚537 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1002263067473920 trainer/global_step‚6549 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚995337651478528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚995337634439168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚997134002582482 $ _timestamp‚1.784278387127785e+09 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚990218778963968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11475948™ mˆÓ2ó- _wandbruntime‚5579  _runtime‚5579þ©ˆö&ôðM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226373452924416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚995484875468800 trainer/global_step‚6549 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1002411679516160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚995484892508160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916217632988 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25585240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60664261 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚997281524308188 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚538 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚995484875468800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚5579.865402449 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11477693 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7002996860416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112606002 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1002411679516160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚995484892508160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51931748 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6926787008000 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1002411321130496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226373347839488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚990365306675200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7003083333120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60674252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60674252 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112596009 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6926787008000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚997281166718826 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25584976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14107283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6915877082986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60664261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219370369591296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842783877395704e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14107544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51931750 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25584976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14107283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6915877082986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60664261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219370369591296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842783877395704e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14107544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51931750 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11477693 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7002996860416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112606002 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1002411679516160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚995484892508160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51931748 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6926787008000 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1002411321130496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226373347839488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0š–ô# 2ö- _wandbruntime‚5579  _runtime‚5579ÑQ[9 ÷„ 2loss/FBt1/assignment_loss‚0.003624079590672242 1loss/SRqqt1/detection_loss‚0.9391617838008014 $loss/total_loss‚8.29330459808282 _step‚539 2loss/SRqqt1/assignment_loss‚0.4646798344752576 /loss/FRt1/detection_loss‚0.5021307819537706 /loss/FBt1/detection_loss‚0.5068598703417758 epoch‚16 _runtime‚5579.868219797 0loss/FRt1/assignment_loss‚1.7647885401553638 4loss/FBt2/assignment_loss‚0.00047438602495343795 .loss/FRt2/detection_loss‚1.270650937331097 /loss/FBt2/detection_loss‚0.7929518063794132 trainer/global_step‚6549 0loss/FRt2/assignment_loss‚0.9531577481042355 3loss/SRqqt2/assignment_loss‚0.49640854480535274 1loss/SRqqt2/detection_loss‚0.5984162851201259 $ _timestamp‚1.784278387742802e+09fW[t×ø‰ /loss/FBt2/detection_loss‚0.7929518063794132 trainer/global_step‚6549 0loss/FRt2/assignment_loss‚0.9531577481042355 3loss/SRqqt2/assignment_loss‚0.49640854480535274 1loss/SRqqt2/detection_loss‚0.5984162851201259 $ _timestamp‚1.784278387742802e+09 2loss/FBt1/assignment_loss‚0.003624079590672242 1loss/SRqqt1/detection_loss‚0.9391617838008014 $loss/total_loss‚8.29330459808282 _step‚539 2loss/SRqqt1/assignment_loss‚0.4646798344752576 /loss/FRt1/detection_loss‚0.5021307819537706 /loss/FBt1/detection_loss‚0.5068598703417758 epoch‚16 _runtime‚5579.868219797 0loss/FRt1/assignment_loss‚1.7647885401553638 4loss/FBt2/assignment_loss‚0.00047438602495343kºÖp8795 .loss/FRt2/detection_loss‚1.270650937331097›¡““ù‚(:ˆ óÚçÒƒÌü£ gpu.0.gpu‚64.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0銰ú‚(:ö óÚçÒ„›Î¦$ disk./.usageGB‚545.6359672546387 disk.sda3.out‚ 192.08203125# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965( disk.nvme0c0n1.out‚27668.08740234375 disk.dm-1.out‚ 192.08203125 disk.dm-0.in‚ 279.4140625# disk.dm-0.out‚27668.08740234375 disk.rbd0.in‚0) disk./.usagePercent‚36.61808758152919 disk.sda.out‚ 192.08203125 disk.sda2.in‚0 disk.rbd3.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 279.4140625$ memory_percent‚95.46947479248047 network.recv‚ 1.4797e+06 disk.sda.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚27668.08740234375 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875 network.sent‚ 1.0416484e+07 disk.sda2.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 279.4140625 cpu‚26.021206760908086 disk.rbd1.out‚97.23193359375% proc.memory.availableMB‚ 742.28125㹯“û‚(:ˆ ‚ÛçÒþŸæ¤ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C¹|^ü‚(:ú ‚ÛçÒ¨ðŠ§ network.recv‚ 1.484342e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚97.23193359375 disk.rbd3.out‚0 disk.sda.out‚ 192.27734375 disk.sda2.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 742.22265625 cpu‚26.02070252099303) disk./.usagePercent‚36.61811446232028$ disk./.usageGB‚545.6363677978516 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2817.87109375 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 1.0481174e+07 disk.dm-1.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚27677.39306640625& disk.nvme0n1.out‚27677.39306640625( disk.nvme0c0n1.out‚27677.39306640625 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 192.27734375 disk.rbd1.in‚ 0.01171875 disk.nvme0n1.in‚ 280.2421875 disk.sda1.out‚0 disk.sda3.out‚ 192.27734375$ memory_percent‚95.46983242034912 disk.dm-0.in‚ 280.2421875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 280.2421875* proc.memory.percent‚17.198920249938965‹h[“ý‚(:ˆ ‘ÛçÒœŒä¤ gpu.0.gpu‚96.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚248.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›~UOþ‚(:ú ‘ÛçÒýží¦ disk.sda1.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 280.2421875( disk.nvme0c0n1.out‚27681.02783203125 disk.sda2.out‚0 disk.sda3.out‚ 192.47265625$ memory_percent‚95.46992778778076 network.sent‚ 1.0486186e+07 network.recv‚ 1.487866e+06# disk.dm-0.out‚27681.02783203125 disk.nvme0n1.in‚ 280.2421875 disk.rbd0.out‚0 disk.rbd1.in‚ 0.01171875& disk.nvme0n1.out‚27681.02783203125 disk.sda1.in‚0 disk.sda.out‚ 192.47265625 disk.drbd2165.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 742.20703125 cpu‚26.01989830727357 disk.rbd3.out‚0 disk.dm-0.in‚ 280.2421875 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd1.out‚97.29443359375# proc.memory.rssMB‚ 2817.87109375 disk.rbd3.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.61810447802644$ disk./.usageGB‚545.6362190246582 disk.sda3.in‚0 disk.dm-1.out‚ 192.47265625* proc.memory.percent‚17.198920249938965ÂŹ2ÿ- _wandbruntime‚5619  _runtime‚5619óh^èɀà ' lr-AdamW/pg1‚0.00036529435652749676 ' lr-AdamW/pg2‚0.00036529435652749676 trainer/global_step‚6599 % _timestamp‚1.7842784270613909e+09 _runtime‚5619.186947798 _step‚540eνÎÈ _runtime‚5619.186947798 _step‚540 ' lr-AdamW/pg1‚0.00036529435652749676 ' lr-AdamW/pg2‚0.00036529435652749676 trainer/global_step‚6599 % _timestamp‚1.7842784270613909e+09œmå‡2‚- _wandbruntime‚5619  _runtime‚5619妗¹ñ'ƒëO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25773405 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61119638 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009675814457344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1009676172835840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61109661 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002699685920768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1002699685920768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6976128536576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚997544312557568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6965823066062 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1004509795073740 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1002699702960128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61109661 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7053324957184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1009675814457344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009676172835840 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221008378759168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5619.188184082 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6976469875712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52305718 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14209937 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52305716 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221008360146944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7053244768768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11563198 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14210204 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61119638 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228061604915712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842784270624168e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009676172835840 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221008378759168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5619.188184082 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6976469875712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52305718 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14209937 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52305716 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221008360146944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7053244768768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11563198 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14210204 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61119638 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228061604915712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842784270624168e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009675814457344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1009676172835840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61109661 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002699685920768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1002699685920768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6976128536576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚997544312557568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6965823066062 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1004509795073740 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1002699702960128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61109661 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7053324957184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1009675814457344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113432061 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6977482509824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221041788905472 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1004657316799334 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚997690840268800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1009824068113920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52313350 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61118713 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7054265605120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6966834119896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25776961 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1002846926950400 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113442054 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚ 5619.79942169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52313348 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚542 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228096054510592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7054349980672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25777213 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1009824426499584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144†]Ý'‡úM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1002846926950400 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113442054 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚ 5619.79942169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52313348 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚542 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228096054510592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7054349980672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25777213 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1009824426499584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52313348 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1002846943989760 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1002846926950400 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61128704 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1002846943989760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113432061 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6977482509824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221041788905472 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1004657316799334 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚997690840268800 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1009824068113920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52313350 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61118713 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7054265605120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6966834119896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25776961 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ž3Ãã2ˆ- _wandbruntime‚5619  _runtime‚5619½¿0‰€ 3loss/SRqqt2/assignment_loss‚0.46620222509533443 2loss/FBt1/assignment_loss‚0.012082673389231266 /loss/FRt1/detection_loss‚0.5469541395064226 epoch‚16 % _timestamp‚1.7842784276768239e+09 1loss/SRqqt1/detection_loss‚0.9719801533350406 "loss/total_loss‚8.095457614201 _runtime‚5619.802146875 _step‚543 1loss/SRqqt2/detection_loss‚0.6673438036820196 2loss/FBt2/assignment_loss‚0.009760630544907802 .loss/FRt2/detection_loss‚1.209694578582551 /loss/FBt1/detection_loss‚0.4700609504652675 .loss/FBt2/detection_loss‚0.626246228501008 trainer/global_step‚6599 0loss/FRt1/assignment_loss‚1.6908940571671416 0loss/FRt2/assignment_loss‚0.9190046799744611 2loss/SRqqt1/assignment_loss‚0.5052334939576161íû¨L Š… 2loss/SRqqt1/assignment_loss‚0.5052334939576161 3loss/SRqqt2/assignment_loss‚0.46620222509533443 2loss/FBt1/assignment_loss‚0.012082673389231266 /loss/FRt1/detection_loss‚0.5469541395064226 epoch‚16 % _timestamp‚1.7842784276768239e+09 1loss/SRqqt1/detection_loss‚0.9719801533350406 "loss/total_loss‚8.095457614201 _runtime‚5619.802146875 _step‚543 1loss/SRqqt2/detection_loss‚0.6673438036820196 2loss/FBt2/assignment_loss‚0.009760630544907802 .loss/FRt2/detection_loss‚1.209694578582551 /loss/FBt1/detection_loss‚0.4700609504652675 .loss/FBt2/detection_loss‚0.626246228501008 trainer/global_step‚6599 0loss/FRt1/assignment_loss‚1.6908940571671416 0loss/FRt2/assignment_loss‚0.9190046799744611Ÿè¢!ú“‹‚(:ˆ  ÛçÒ™…£ gpu.0.gpu‚90.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚159.987' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F’þ¥Œ‚(:÷  ÛçÒË¥¬ disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.0489209e+07 disk.dm-1.out‚ 192.6484375 disk.rbd1.out‚97.29443359375 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚27690.85009765625$ disk./.usageGB‚545.6360931396484" disk.nvme0c0n1.in‚ 280.95703125 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 280.95703125& disk.nvme0n1.out‚27690.85009765625$ memory_percent‚95.47138214111328% proc.memory.availableMB‚ 741.96875) disk./.usagePercent‚36.61809602977782 disk.sda.out‚ 192.6484375* proc.memory.percent‚17.198920249938965 cpu‚26.01908810060279 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚27690.85009765625 disk.sda3.out‚ 192.6484375 disk.rbd1.in‚ 0.01171875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2817.87109375 network.recv‚ 1.491109e+06 disk.nvme0n1.in‚ 280.95703125 disk.sda1.in‚0ع6­ž‚(:“ ¯ÛçÒ —å¤ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:É)Ž‚(:ö ¯ÛçÒ°ðò¦ network.recv‚ 1.496025e+06& disk.nvme0c0n1.out‚27699.033203125 disk.dm-1.out‚ 192.98046875$ disk.nvme0n1.out‚27699.033203125 disk.sda2.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.198920249938965$ disk./.usageGB‚545.6361427307129 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 281.78515625$ memory_percent‚95.46947479248047 cpu‚26.018341825604537 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd1.out‚97.35693359375! disk.dm-0.out‚27699.033203125# proc.memory.rssMB‚ 2817.87109375 network.sent‚ 1.0552251e+07* disk./.usagePercent‚36.618099357875764 disk.sda1.in‚0 disk.rbd1.in‚ 0.01171875% proc.memory.availableMB‚ 742.28125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 192.98046875" disk.nvme0c0n1.in‚ 281.78515625 disk.sda2.out‚0 disk.sda3.out‚ 192.98046875 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 281.78515625·ß Ï“‚(:ˆ ¾ÛçÒÓ€„¥ gpu.0.gpu‚38.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CiÊñ‚(:æ ¾ÛçÒ„§ disk.rbd0.out‚0# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 disk.rbd3.in‚0" disk.dm-0.out‚27708.7314453125' disk.nvme0c0n1.out‚27708.7314453125 network.sent‚ 1.0556338e+07) disk./.usagePercent‚36.61799874691481 disk.dm-1.out‚ 193.19140625 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 193.19140625 disk.rbd1.out‚97.35693359375$ memory_percent‚95.46945095062256 cpu‚26.017502953443504 disk.rbd1.in‚ 0.01171875 disk.nvme0c0n1.in‚282.5$ disk./.usageGB‚545.6346435546875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 193.19140625 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚282.5 disk.sda1.in‚0( proc.memory.availableMB‚ 742.28515625 disk.dm-1.in‚0 disk.dm-0.in‚282.5% disk.nvme0n1.out‚27708.7314453125 disk.sda1.out‚0 network.recv‚ 1.498317e+06 disk.sda2.out‚0 disk.rbd0.in‚0ÛlV#2‘- _wandbruntime‚5659  _runtime‚5659¬áȌɒà ' lr-AdamW/pg1‚0.00036370302144524467 ' lr-AdamW/pg2‚0.00036370302144524467 trainer/global_step‚6649 % _timestamp‚1.7842784669521143e+09 _runtime‚5659.077703993 _step‚544÷ÀÎ+Î“È _runtime‚5659.077703993 _step‚544 ' lr-AdamW/pg1‚0.00036370302144524467 ' lr-AdamW/pg2‚0.00036370302144524467 trainer/global_step‚6649 % _timestamp‚1.7842784669521143e+09 Gõýü2”- _wandbruntime‚5659  _runtime‚5659«Bhœô'•îO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7104575677952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚229784392363520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52687316 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1017088561440768 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1010061754441728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14314636 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1017088919819264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61564113 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚545 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52687318 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7026824038400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7026824038400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222679816685568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7016439552970 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7016099003080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25965368 _runtime‚5659.078957229 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1017088919819264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11650448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136Š7<Ä+ –óO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1017088561440768 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1010061754441728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14314636 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1017088919819264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61564113 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚545 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52687318 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7026824038400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7026824038400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222679816685568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7016439552970 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7016099003080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25965368 _runtime‚5659.078957229 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1017088919819264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11650448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7104575677952 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚229784392363520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52687316 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977¡3‚d¶2—- _wandbruntime‚5659  _runtime‚5659’Õvôþ&˜øM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7105505264640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1012033466879842 % _timestamp‚1.7842784675648289e+09 _step‚546 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1017237173483008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61583156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52694950 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222713226831872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7027836665344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229818732096512 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114278106 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1010208978432000 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1017237173483008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52694950 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14316731 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1017236815097344 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25968924 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1010208995471360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7017450606804 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7027836665344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7028178011648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114268113 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114278106 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114278106 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7105505264640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1012033466879842 % _timestamp‚1.7842784675648289e+09 _step‚546 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1017237173483008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61583156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52694950 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222713226831872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7027836665344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229818732096512 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114278106 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1010208978432000 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1017237173483008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52694950 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14316731 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1017236815097344 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25968924 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1010208995471360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7017450606804 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7027836665344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7028178011648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114268113 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F›€ 0loss/FRt1/assignment_loss‚1.6850804808312023 /loss/FRt2/assignment_loss‚0.991818519747048 /loss/FRt2/detection_loss‚1.2252973052021525 /loss/FBt2/detection_loss‚1.1710881072406967 trainer/global_step‚6649 % _timestamp‚1.7842784675676875e+09 2loss/SRqqt2/assignment_loss‚0.4741571296388583 1loss/SRqqt1/detection_loss‚0.9520931239674488 %loss/total_loss‚8.804302253813749 epoch‚16 _step‚547 /loss/FBt1/assignment_loss‚0.033320888364527 /loss/FRt1/detection_loss‚0.5486030082027573 1loss/SRqqt2/detection_loss‚0.6395364689435192 3loss/SRqqt1/assignment_loss‚0.49581944860841354 1loss/FBt2/assignment_loss‚0.05011641209665522 /loss/FBt1/detection_loss‚0.5373713609704707 _runtime‚5659.693019871zíPu œ… _step‚547 /loss/FBt1/assignment_loss‚0.033320888364527 /loss/FRt1/detection_loss‚0.5486030082027573 1loss/SRqqt2/detection_loss‚0.6395364689435192 3loss/SRqqt1/assignment_loss‚0.49581944860841354 1loss/FBt2/assignment_loss‚0.05011641209665522 /loss/FBt1/detection_loss‚0.5373713609704707 _runtime‚5659.693019871 0loss/FRt1/assignment_loss‚1.6850804808312023 /loss/FRt2/assignment_loss‚0.991818519747048 /loss/FRt2/detection_loss‚1.2252973052021525 /loss/FBt2/detection_loss‚1.1710881072406967 trainer/global_step‚6649 % _timestamp‚1.7842784675676875e+09 2loss/SRqqt2/assignment_loss‚0.4741571296388583 1loss/SRqqt1/detection_loss‚0.9520931239674488 %loss/total_loss‚8.804302253813749 epoch‚16£ÃqoIž‚(:“ ÍÛçÒê©œ¥ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯,‚ýž‚(:ò ÍÛçÒÛ´±§ network.sent‚ 1.0574396e+07' disk.nvme0c0n1.out‚27716.2607421875 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 1.501688e+06$ memory_percent‚95.46887874603271$ disk./.usageGB‚545.6350479125977 disk.sda3.in‚0 disk.nvme0n1.in‚ 283.328125 disk.rbd3.out‚0 disk.dm-1.out‚ 193.3828125 disk.rbd3.in‚0 disk.dm-0.in‚ 283.328125" disk.dm-0.out‚27716.2607421875 disk.drbd2165.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.198920249938965 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚97.41943359375 disk.sda1.out‚0 cpu‚26.016710017060856 disk.sda.out‚ 193.3828125 disk.sda3.out‚ 193.3828125% disk.nvme0n1.out‚27716.2607421875) disk./.usagePercent‚36.61802588371343 disk.nvme0c0n1.in‚ 283.328125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 742.37890625# proc.memory.rssMB‚ 2817.87109375d¡Ö“Ÿ‚(:ˆ ÜÛçÒÅ­ü¤ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚241.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u‚°Ÿú ‚(:ï ÜÛçÒ¤–§ disk.sda1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 283.328125& disk.nvme0c0n1.out‚27723.697265625$ disk./.usageGB‚545.6362495422363 disk.rbd1.in‚ 0.01171875 disk.nvme0n1.in‚ 283.328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 network.sent‚ 1.0626482e+07 disk.sda1.out‚0 disk.dm-0.in‚ 283.328125$ memory_percent‚95.47040462493896# proc.memory.rssMB‚ 2817.87109375 network.recv‚ 1.506264e+06 disk.rbd3.out‚0 cpu‚26.015928186941462) disk./.usagePercent‚36.61810652608672 disk.rbd0.in‚0 disk.sda3.in‚0! disk.dm-0.out‚27723.697265625( proc.memory.availableMB‚ 742.12890625* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17$ disk.nvme0n1.out‚27723.697265625 disk.sda.out‚ 193.6640625 disk.sda3.out‚ 193.6640625 disk.dm-1.out‚ 193.6640625 disk.rbd1.out‚97.41943359375 G9Ø2¡- _wandbruntime‚5698  _runtime‚5698¦ÚbÇ¢Á _step‚548 & lr-AdamW/pg1‚0.0003621035056956238 & lr-AdamW/pg2‚0.0003621035056956238 trainer/global_step‚6699 % _timestamp‚1.7842785066975887e+09 _runtime‚5698.823181015¦\U Ì£Æ & lr-AdamW/pg2‚0.0003621035056956238 trainer/global_step‚6699 % _timestamp‚1.7842785066975887e+09 _runtime‚5698.823181015 _step‚548 & lr-AdamW/pg1‚0.0003621035056956238¤Þ’2¤- _wandbruntime‚5698  _runtime‚5698æÓ -¥îO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115087481 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7077860879360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14418832 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62018565 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚‹6:ùÇ$ 102994944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224351254611968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7077519540224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017423788883968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7077519540224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7077860879360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1012195396784128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1024501308424192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017423805923328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7155775881728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62018565 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚5698.824537616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11737701 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53068918 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842785066987512e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11737698 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1024501666802688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115097460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1024501666802688 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1024501308424192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26156530 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231507114876416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62028542 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26156793 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231507011881472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224351235999744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53068918 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1019262452824006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1017423788883968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53068916 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7066715489988 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53068916 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7155860264448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438Ž3âù'¦óO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7077519540224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017423788883968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7077519540224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7077860879360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1012195396784128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1024501308424192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017423805923328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7155775881728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62018565 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚5698.824537616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11737701 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53068918 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842785066987512e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11737698 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1024501666802688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115097460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1024501666802688 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1024501308424192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26156530 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231507114876416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62028542 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26156793 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231507011881472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224351235999744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53068918 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1019262452824006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1017423788883968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53068916 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7066715489988 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53068916 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7155860264448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115087481 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7077860879360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14418832 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62018565 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224351254611968¥4¢k_2§- _wandbruntime‚5699  _runtime‚5699€Îð`ý&¨÷M RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115114158 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1017571029913600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚ 5699.43676642 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115104165 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1012341907456000 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7078532167168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53076550 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62037608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224384683370496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7078532167168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7068067093712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚231541465349120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚550 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14421107 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1024649920466432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11739446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224384664758272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7067726543710 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53076550 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14420860 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7156884966400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231541568336896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1019409974549712 % _timestamp‚1.7842785073109505e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53076548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1019409616960350 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26160303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7156800590848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62027617 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26160553 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438•1¼2î ©üM QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚ 5699.43676642 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115104165 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1012341907456000 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7078532167168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53076550 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62037608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224384683370496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7078532167168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7068067093712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚231541465349120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚550 C9DeviceStatsMonitor.otW Šn_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14421107 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1024649920466432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11739446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224384664758272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7067726543710 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53076550 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14420860 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7156884966400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231541568336896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1019409974549712 % _timestamp‚1.7842785073109505e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53076548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1019409616960350 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26160303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7156800590848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62027617 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26160553 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115114158 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1017571029913600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0¦¶V'!2ª- _wandbruntime‚5699  _runtime‚5699(Có»«‚ 1loss/SRqqt2/detection_loss‚0.6884933261339441 /loss/FBt2/detection_loss‚1.0269434478666102 $loss/total_loss‚8.66214247876542 trainer/global_step‚6699 $ _timestamp‚1.784278507313879e+09 3loss/SRqqt1/assignment_loss‚0.49935318242745175 1loss/SRqqt1/detection_loss‚1.0276071792895285 0loss/FBt1/detection_loss‚0.39179383667094486 /loss/FRt1/assignment_loss‚1.686272243459355 2loss/SRqqt2/assignment_loss‚0.4400578859735521 1loss/FBt2/assignment_loss‚0.01508997686392386 /loss/FRt1/detection_loss‚0.5449950258723971 epoch‚16 _step‚551 0loss/FRt2/assignment_loss‚0.9698047077148988 /loss/FRt2/detection_loss‚1.3564857208207008 _runtime‚5699.439231884 2loss/FBt1/assignment_loss‚0.015245945672112278+²û ¬‡ /loss/FRt1/assignment_loss‚1.686272243459355 2loss/SRqqt2/assignment_loss‚0.4400578859735521 1loss/FBt2/assignment_loss‚0.01508997686392386 /loss/FRt1/detection_loss‚0.5449950258723971 epoch‚16 _step‚551 0loss/FRt2/assignment_loss‚0.9698047077148988 /loss/FRt2/detection_loss‚1.3564857208207008 _runtime‚5699.439231884 2loss/FBt1/assignment_loss‚0.015245945672112278 1loss/SRqqt2/detection_loss‚0.6884933261339441 /loss/FBt2/detection_loss‚1.0269434478666102 $loss/total_loss‚8.66214247876542 trainer/global_step‚6699 $ _timestamp‚1.784278507313879e+09 3loss/SRqqt1/assignment_loss‚0.49935318242745175 1loss/SRqqt1/detection_loss‚1.0276071792895285 0loss/FBt1/detection_loss‚0.39179383667094486§³-M­‚(:… ëÛçÒŸßŤ gpu.0.gpu‚92.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.23 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L:Ϥ ®‚(:þ ëÛçÒ‚ É¦$ memory_percent‚96.97320461273193( proc.memory.availableMB‚ 495.91015625 network.sent‚ 1.0629505e+07 disk.sda3.in‚0 cpu‚26.015194705045595 disk.rbd3.out‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 284.04296875 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2817.87109375) disk./.usagePercent‚36.61818614442986 disk.nvme0n1.in‚ 284.04296875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚27731.06201171875 disk.dm-1.in‚0 disk.rbd1.out‚97.48193359375# disk.dm-0.out‚27731.06201171875 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda3.out‚ 194.26953125* proc.memory.percent‚17.198920249938965 network.recv‚ 1.509689e+06$ disk./.usageGB‚545.6374359130859 disk.dm-1.out‚ 194.26953125 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 284.04296875& disk.nvme0n1.out‚27731.06201171875 disk.drbd2165.out‚0 disk.sda.out‚ 194.26953125SÞWZž¯‚(:“ úÛçÒïëÓ¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.079' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.015800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æõ¸ü°‚(:ñ úÛç҇匧 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.dm-0.out‚ 27742.2890625 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 27742.2890625 network.sent‚ 1.0694635e+07 disk.sda.in‚0* proc.memory.percent‚17.198920249938965) disk./.usagePercent‚36.61817539211343 disk.rbd0.in‚0 disk.sda.out‚ 194.48046875 disk.rbd3.out‚0" disk.nvme0n1.out‚ 27742.2890625 disk.sda1.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚97.48193359375# proc.memory.rssMB‚ 2817.87109375 cpu‚26.01441225401068 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 284.87109375" disk.nvme0c0n1.in‚ 284.87109375 disk.dm-0.in‚ 284.87109375 network.recv‚ 1.514528e+06$ disk./.usageGB‚545.6372756958008 disk.dm-1.out‚ 194.48046875$ memory_percent‚96.97163105010986 disk.sda2.out‚0 disk.sda3.out‚ 194.48046875( proc.memory.availableMB‚ 496.16796875 proc.cpu.threads‚17Bˆ ž±‚(:“ ‰ÜçÒ¬Ô•¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚280.079' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.015800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ýiMö²‚(:ë ‰ÜçÒü¸“¦ disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 194.6953125 disk.rbd1.out‚97.54443359375* proc.memory.percent‚17.198920249938965 disk.nvme0n1.in‚ 285.5859375 disk.drbd2165.in‚0 disk.dm-0.out‚ 27750.5390625$ disk./.usageGB‚545.6375274658203 disk.sda3.in‚0# proc.memory.rssMB‚ 2817.87109375 disk.sda1.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.0699687e+07 network.recv‚ 1.518231e+06" disk.nvme0n1.out‚ 27750.5390625 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 27750.5390625 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.61819228861069 disk.sda.out‚ 194.6953125$ memory_percent‚96.97210788726807( proc.memory.availableMB‚ 496.08984375 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 285.5859375 cpu‚26.01369122121206 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 285.5859375 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 194.6953125`-t¨2³- _wandbruntime‚5738  _runtime‚5738?ƒÉÉ´Ã ' lr-AdamW/pg1‚0.00036049591201254145 ' lr-AdamW/pg2‚0.00036049591201254145 trainer/global_step‚6749 % _timestamp‚1.7842785466430116e+09 _runtime‚5738.768599033 _step‚552« „ÁÎµÈ _step‚552 ' lr-AdamW/pg1‚0.00036049591201254145 ' lr-AdamW/pg2‚0.00036049591201254145 trainer/global_step‚6749 % _timestamp‚1.7842785466430116e+09 _runtime‚5738.768599033¨ü´ßw2¶- _wandbruntime‚5738  _runtime‚5738…R4aô'·îO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14522624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842785466442692e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚553 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53450516 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14522358 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1019520913338368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚5738.770347013 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62482994 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26347575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1024785857404928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1019520930377728 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11824951 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62473017 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7128556381184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115933512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226022673926144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1024785857404928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7128556381184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233229898466816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1031914055407616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226022692538368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438yÛÏc#¸óO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26347575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1024785857404928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1019520930377728 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11824951 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62473017 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7128556381184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115933512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226022673926144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1024785857404928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7128556381184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233229898466816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1031914055407616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226022692538368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14522624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842785466442692e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚553 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53450516 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14522358 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1019520913338368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚5738.770347013 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62482994 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438©É2Þ2¹- _wandbruntime‚5739  _runtime‚5739c-Æý&º÷M LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62482069 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14524473 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115950210 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11826696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129569015296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233264254428160 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53458148 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5739.387482467 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1032062309064192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1024933098434560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7118683580620 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1024933081395200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53458148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7208227730432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53458150 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115950210 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115940217 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1024933081395200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129569015296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62492060 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62482069 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚233264349027328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1032062667449856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115940217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62492060 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226056102684672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53458150 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26351166 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1024933098434560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129227668992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1026786124630220 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26351426 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7842785472616384e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976y4Z'»üM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚5739.387482467 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1032062309064192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1024933098434560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7118683580620 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1024933081395200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53458148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7208227730432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53458150 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115950210 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115940217 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1024933081395200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129569015296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62492060 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62482069 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚233264349027328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1032062667449856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115940217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62492060 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚226056102684672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53458150 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26351166 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1024933098434560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129227668992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1026786124630220 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26351426 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7842785472616384e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62482069 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14524473 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115950210 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11826696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129569015296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233264254428160 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53458148 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1034014752985022 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62937446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227694130464768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227694111852544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7258515278848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7167948463804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62927469 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1039327160769536 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1039326802391040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7178910543872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53832116 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7178910543872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116769564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11912201 % _timestamp‚1.7842785867073653e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1039327160769536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62927469 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034014395395772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚557 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1032147908886528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234952627131392 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227694130464768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227694111852544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7258515278848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7167948463804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62927469 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1039327160769536 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1039326802391040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7178910543872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53832116 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7178910543872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116769564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11912201 % _timestamp‚1.7842785867073653e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1039327160769536 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62927469 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034014395395772 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚557 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1032147908886528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234952627131392 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1034014752985022 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62937446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚558 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53839748 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _runtime‚5779.445781687 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1032295132876800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62946512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234987078204928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1032295132876800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116776269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6799 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1032295149916160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26542666 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116786262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62936521 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53839750 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62936521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11913943 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7259626163712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116786262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53839748 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14628720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1026992957603840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1032295149916160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26542382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227727540611072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7842785873199985e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14628439 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7168959517526 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7179923170816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1039475056047616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7180264517120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚234987185387008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1034161917121366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1026992974643200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7169300067528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11913946 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1039475414433280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7180264517120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227727559223296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62946512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7179923170816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1032295149916160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26542382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227727540611072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7842785873199985e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14628439 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7168959517526 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7179923170816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1039475056047616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7180264517120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚234987185387008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1034161917121366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1026992974643200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7169300067528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11913946 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1039475414433280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7180264517120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227727559223296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62946512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7179923170816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚558 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53839748 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _runtime‚5779.445781687 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1032295132876800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62946512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234987078204928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FD´ÌÉ ` eviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1032295132876800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116776269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6799 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1032295149916160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26542666 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116786262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62936521 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53839750 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62936521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11913943 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7259626163712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116786262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53839748 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14628720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616®| Ï2Î- _wandbruntime‚5779  _runtime‚5779È]P σ 2loss/SRqqt2/assignment_loss‚0.5673172909854178 1loss/FBt2/assignment_loss‚0.06330113848699769 1loss/SRqqt1/detection_loss‚0.9030766447785975 /loss/FBt2/detection_loss‚0.7648569321946094 trainer/global_step‚6799 /loss/FRt2/assignment_loss‚0.945954426575233 3loss/SRqqt1/assignment_loss‚0.46128685854646506 2loss/FBt1/assignment_loss‚0.016287395575494152 /loss/FRt1/detection_loss‚0.5138551755965745 /loss/FRt2/detection_loss‚1.3016911093669905 %loss/total_loss‚8.309950068385668 % _timestamp‚1.7842785873232734e+09 1loss/SRqqt2/detection_loss‚0.6203166353777989 0loss/FBt1/detection_loss‚0.40877498111593824 epoch‚16 _runtime‚ 5779.44860591 0loss/FRt1/assignment_loss‚1.7432314797855508 _step‚559øsdÜЈ 0loss/FRt1/assignment_loss‚1.7432314797855508 _step‚559 2loss/SRqqt2/assignment_loss‚0.5673172909854178 1loss/FBt2/assignment_loss‚0.06330113848699769 1loss/SRqqt1/detection_loss‚0.9030766447785975 /loss/FBt2/detection_loss‚0.7648569321946094 trainer/global_step‚6799 /loss/FRt2/assignment_loss‚0.945954426575233 3loss/SRqqt1/assignment_loss‚0.46128685854646506 2loss/FBt1/assignment_loss‚0.016287395575494152 /loss/FRt1/detection_loss‚0.5138551755965745 /loss/FRt2/detection_loss‚1.3016911093669905 %loss/total_loss‚8.309950068385668 % _timestamp‚1.7842785873232734e+09 1loss/SRqqt2/detection_loss‚0.6203166353777989 0loss/FBt1/detection_loss‚0.40877498111593824 epoch‚16 _runtime‚ 5779.44860591¯ZNÎB“Ñ‚(:ˆ ÅÜçÒ§¨¯¢ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 yÕÒ‚(:ú ÅÜçÒßÁš§ disk.drbd2165.out‚0 disk.sda.out‚ 201.29296875 disk.sda2.in‚0 disk.dm-1.out‚ 201.29296875 disk.rbd3.in‚0# proc.memory.rssMB‚ 2817.87109375 disk.nvme0n1.in‚ 288.0078125 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.532342e+06( disk.nvme0c0n1.out‚27811.84912109375 disk.rbd3.out‚0$ memory_percent‚96.97449207305908 cpu‚26.01761044513434 network.sent‚ 1.0776014e+07 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚97.66943359375# disk.dm-0.out‚27811.84912109375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 495.69921875$ disk./.usageGB‚545.6362533569336& disk.nvme0n1.out‚27811.84912109375! disk.nvme0c0n1.in‚ 288.0078125 disk.sda.in‚0) disk./.usagePercent‚36.61810678209425 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 288.0078125* proc.memory.percent‚17.198920249938965 disk.drbd2165.in‚0 disk.sda3.out‚ 201.29296875 proc.cpu.threads‚17 nˆ2Ó- _wandbruntime‚5796  _runtime‚5796šq|ÐÖ=ÔÐ{ @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972619414329529 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6282604932785034 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7796162962913513 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5907954573631287 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5080735683441162 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_3‚0.9285281896591187 +jet/accuracy_3_of_4‚0.08834609389305115 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6301572322845459 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_3_of_4‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2994314432144165 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.647699236869812 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6784499287605286 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7060515880584717 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02968566119670868 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7586705684661865 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15651735663414001 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8624163866043091 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7503988146781921 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03241117298603058 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0852782279253006 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.14814814925193787 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30663731694221497 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6643602252006531 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6697548031806946 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.40372008085250854 +jet/accuracy_4_of_4‚0.00957049522548914 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2390761375427246 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5104036927223206 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6909068822860718 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6948654055595398 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6750891804695129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5567973256111145 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993925929069519 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.716675877571106 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16277362406253815 (particle/accuracy‚0.8429479002952576 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_2‚0.8140385746955872 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9030560255050659 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2662211060523987 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3609124422073364 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9875140190124512 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.054078392684459686 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7495490312576294 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043156784027814865 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29432159662246704 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23826663196086884 _runtime‚5796.542738732 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7768329381942749 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_4‚0.9675389528274536 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.398929238319397 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2586991488933563 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7417044639587402 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14306508004665375 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7842786044165049e+09 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44083040952682495 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6549144387245178 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6517504453659058 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5834014415740967 +particle/sensitivity‚0.6231276392936707 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4003276228904724 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7043583989143372 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2260434925556183 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7930461168289185 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2244323045015335 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 -"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4375 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 epoch‚16 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3171013295650482 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933705925941467 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23353850841522217 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049726974219083786 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7997847199440002 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7289938926696777 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 *jet/accuracy_2_of_4‚0.8044496178627014 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5094254612922668 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3185255527496338 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8479262590408325 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995890736579895 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26966410875320435 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15971317887306213 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7990292310714722 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚6799 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6720549464225769 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3333333134651184 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2791179418563843 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25482654571533203 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2549065947532654 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6756613850593567 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6566706895828247 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 /particle/accuracy_4_of_4‚0.9412819147109985 _step‚560 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7956160306930542 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7741926312446594 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7558587193489075 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877747893333435 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4966222643852234 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 +jet/accuracy_1_of_1‚0.36749377846717834 particle/accuracy_5_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_2_of_5‚1 )loss/val_total_loss‚8.306898105649198 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_1_of_1‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9028980731964111 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7540543079376221 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_3_of_3‚0.0693003311753273 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9872700572013855 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5803542137145996 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9840445518493652 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7078425884246826 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3090658187866211 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3090658187866211 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967955350875854 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.495133638381958 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7006238102912903 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29491734504699707 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996802806854248 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7569072246551514 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7466275691986084 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9162211418151855 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023932788521051407 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17340858280658722 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22189033031463623 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7481592893600464 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9918553233146667 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03179048374295235 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944580793380737 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030104804784059525 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6720549464225769 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3688930869102478 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7137811183929443 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04459802433848381 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6445210576057434 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5449999570846558 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.855889618396759 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7548089027404785 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24232006072998047 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7549305558204651 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6650510430335999 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_4‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8479262590408325 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023932788521051407 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09518177062273026 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961074590682983 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3191421926021576 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33631768822669983 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8036659955978394 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1å×x+ÕÕ{ +jet/accuracy_1_of_1‚0.36749377846717834 particle/accuracy_5_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_2_of_5‚1 )loss/val_total_loss‚8.306898105649198 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_1_of_1‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9028980731964111 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7540543079376221 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_3_of_3‚0.0693003311753273 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9872700572013855 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5803542137145996 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9840445518493652 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7078425884246826 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3090658187866211 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3090658187866211 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967955350875854 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.495133638381958 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7006238102912903 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29491734504699707 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996802806854248 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7569072246551514 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7466275691986084 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9162211418151855 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023932788521051407 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17340858280658722 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22189033031463623 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7481592893600464 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9918553233146667 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03179048374295235 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944580793380737 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030104804784059525 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6720549464225769 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3688930869102478 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7137811183929443 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04459802433848381 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6445210576057434 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5449999570846558 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.855889618396759 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7548089027404785 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24232006072998047 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7549305558204651 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6650510430335999 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_4‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8479262590408325 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023932788521051407 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09518177062273026 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961074590682983 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3191421926021576 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33631768822669983 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8036659955978394 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972619414329529 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6282604932785034 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7796162962913513 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5907954573631287 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5080735683441162 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_3‚0.9285281896591187 +jet/accuracy_3_of_4‚0.08834609389305115 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6301572322845459 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_3_of_4‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2994314432144165 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.647699236869812 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6784499287605286 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7060515880584717 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02968566119670868 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7586705684661865 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15651735663414001 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8624163866043091 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7503988146781921 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03241117298603058 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0852782279253006 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.14814814925193787 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30663731694221497 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6643602252006531 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6697548031806946 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.40372008085250854 +jet/accuracy_4_of_4‚0.00957049522548914 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2390761375427246 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5104036927223206 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6909068822860718 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6948654055595398 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6750891804695129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5567973256111145 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993925929069519 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.716675877571106 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16277362406253815 (particle/accuracy‚0.8429479002952576 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_2‚0.8140385746955872 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9030560255050659 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2662211060523987 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3609124422073364 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9875140190124512 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.054078392684459686 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7495490312576294 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043156784027814865 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29432159662246704 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23826663196086884 _runtime‚5796.542738732 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7768329381942749 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901½p´Îc786961592734 *jet/accuracy_1_of_4‚0.9675389528274536 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.398929238319397 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2586991488933563 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7417044639587402 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14306508004665375 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7842786044165049e+09 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44083040952682495 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6549144387245178 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6517504453659058 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5834014415740967 +particle/sensitivity‚0.6231276392936707 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4003276228904724 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7043583989143372 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2260434925556183 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7930461168289185 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2244323045015335 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 -"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4375 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 epoch‚16 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3171013295650482 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933705925941467 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23353850841522217 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049726974219083786 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7997847199440002 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7289938926696777 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 *jet/accuracy_2_of_4‚0.8044496178627014 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5094254612922668 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3185255527496338 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8479262590408325 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.995890736579895 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26966410875320435 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15971317887306213 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7990292310714722 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚6799 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6720549464225769 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3333333134651184 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2791179418563843 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25482654571533203 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2549065947532654 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6756613850593567 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6566706895828247 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 /particle/accuracy_4_of_4‚0.9412819147109985 _step‚560 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7956160306930542 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7741926312446594 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7558587193489075 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877747893333435 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4966222643852234 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184°NÍ…Ö‚(:’ ÔÜçÒÇ›‹¥ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚238.658' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚47.73159999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SV¤nýׂ(:ò ÔÜçÒáô”§ disk.sda3.out‚ 201.609375 disk.drbd2165.out‚0* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.sent‚ 1.0852424e+07" disk.nvme0c0n1.in‚ 288.72265625 disk.sda.out‚ 201.609375 disk.rbd1.out‚97.66943359375% disk.nvme0n1.out‚28068.5048828125 disk.sda2.in‚0 disk.dm-1.out‚ 201.609375$ memory_percent‚95.47481536865234$ disk./.usageGB‚545.6373023986816 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 network.recv‚ 1.536505e+06* disk./.usagePercent‚36.618177184166164 disk.rbd0.out‚0' disk.nvme0c0n1.out‚28068.5048828125 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚28068.5048828125 disk.sda1.out‚0 disk.dm-0.in‚ 288.72265625 disk.nvme0n1.in‚ 288.72265625 disk.sda1.in‚0% proc.memory.availableMB‚ 741.40625# proc.memory.rssMB‚ 2817.87109375 cpu‚26.02055685583318Æbg“Ø‚(:ˆ ãÜçÒïóĤ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚246.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.3466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09bÈÙ‚(:ý ãÜçÒ‡¢Ð¦( disk.nvme0c0n1.out‚28127.52392578125( proc.memory.availableMB‚ 741.44921875" disk.nvme0c0n1.in‚ 288.72265625 disk.dm-1.in‚0 disk.dm-1.out‚ 201.95703125# proc.memory.rssMB‚ 2817.87109375 disk.dm-0.in‚ 288.72265625 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0( disk./.usagePercent‚36.6181108782148 disk.rbd1.out‚97.73193359375 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚28127.52392578125 cpu‚26.019827323286606 network.sent‚ 1.0898615e+07 network.recv‚ 1.541204e+06$ disk./.usageGB‚545.6363143920898 disk.drbd2165.in‚0* proc.memory.percent‚17.198920249938965 disk.sda3.out‚ 201.95703125 disk.nvme0n1.in‚ 288.72265625& disk.nvme0n1.out‚28127.52392578125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 201.95703125 disk.sda2.out‚0$ memory_percent‚95.47455310821533 disk.rbd1.in‚ 0.01171875­b¥ª‘Ú‚(:† òÜçÒó Û¤ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.26' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!@)€Û‚(:ö òÜçÒ…ÄŠ§$ disk./.usageGB‚545.6364936828613 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 202.1328125) disk./.usagePercent‚36.61812291056891" disk.nvme0c0n1.in‚ 289.55078125 disk.sda.out‚ 202.1328125 cpu‚26.01904991266308 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚97.73193359375 network.recv‚ 1.544398e+06 disk.nvme0n1.in‚ 289.55078125% disk.nvme0n1.out‚28325.0654296875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-0.in‚ 289.55078125 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 202.1328125# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 disk.sda1.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚28325.0654296875( proc.memory.availableMB‚ 741.44921875' disk.nvme0c0n1.out‚28325.0654296875 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 1.090363e+07$ memory_percent‚95.47455310821533Äv}Í2Ü- _wandbruntime‚5837  _runtime‚5837òwþÐÇÝÁ ' lr-AdamW/pg2‚0.00035725690436914497 trainer/global_step‚6849 $ _timestamp‚1.784278645756674e+09 _runtime‚ 5837.88223182 _step‚561 ' lr-AdamW/pg1‚0.00035725690436914497FãÌÞÆ _runtime‚ 5837.88223182 _step‚561 ' lr-AdamW/pg1‚0.00035725690436914497 ' lr-AdamW/pg2‚0.00035725690436914497 trainer/global_step‚6849 $ _timestamp‚1.784278645756674e+09±°ä‡2ß- _wandbruntime‚5837  _runtime‚5837)¿jÍô'àîO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54334382 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7247339253760 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236709407784448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7327842910720 _step‚562 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12001968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1048686849232896 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1041439527018496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7247680592896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1036080660779008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7236284675946 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7236625225836 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54334384 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63462225 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117796609 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1048686849232896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1048687207611392 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1041439509979136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842786457576666e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12001971 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1043317286004844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1041439509979136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26763784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229381461878784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54334382 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12001968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1048686849232896 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1041439527018496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7247680592896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1036080660779008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7236284675946 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7236625225836 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54334384 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63462225 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117796609 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1048686849232896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1048687207611392 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1041439509979136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842786457576666e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12001971 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1043317286004844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1041439509979136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26763784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229381461878784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54334382 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6849 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7328953773056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229414890637312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1048835461275136 PBDeviceStatsMonitor.on_train_batch_end/active_òW®g¥bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63471291 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚563 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1041586751008768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117813307 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26767331 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1043464807730550 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1041586768048128 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7248351880704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7248693227008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26767588 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ñí+'äûM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1043464450141188 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6849 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7328953773056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229414890637312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1048835461275136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63471291 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚563 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1041586751008768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117813307 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26767331 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1043464807730550 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1041586768048128 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7248351880704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7248693227008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26767588³¿;S}2å- _wandbruntime‚5838  _runtime‚5838“1K|æ‚ epoch‚17 1loss/SRqqt1/detection_loss‚0.9545963993528858 0loss/SRqqt2/detection_loss‚0.717549280237822 %loss/total_loss‚8.409816788642448 2loss/SRqqt2/assignment_loss‚0.5405087350123301 /loss/FRt2/detection_loss‚1.2581464200644406 trainer/global_step‚6849 $ _timestamp‚1.784278646373322e+09 _runtime‚5838.498644455 .loss/FRt1/assignment_loss‚1.72644536584713 0loss/FRt2/assignment_loss‚0.8279934498880591 2loss/SRqqt1/assignment_loss‚0.6294161266804043 /loss/FRt1/detection_loss‚0.5219001178076597 /loss/FBt2/detection_loss‚0.7564844563769709 _step‚564 2loss/FBt1/assignment_loss‚0.017684369675696997 3loss/FBt2/assignment_loss‚0.0005328788546259082 0loss/FBt1/detection_loss‚0.45855918884442387a!*Ï ç‡ 2loss/SRqqt2/assignment_loss‚0.5405087350123301 /loss/FRt2/detection_loss‚1.2581464200644406 trainer/global_step‚6849 $ _timestamp‚1.784278646373322e+09 _runtime‚5838.498644455 .loss/FRt1/assignment_loss‚1.72644536584713 0loss/FRt2/assignment_loss‚0.8279934498880591 2loss/SRqqt1/assignment_loss‚0.6294161266804043 /loss/FRt1/detection_loss‚0.5219001178076597 /loss/FBt2/detection_loss‚0.7564844563769709 _step‚564 2loss/FBt1/assignment_loss‚0.017684369675696997 3loss/FBt2/assignment_loss‚0.0005328788546259082 0loss/FBt1/detection_loss‚0.45855918884442387 epoch‚17 1loss/SRqqt1/detection_loss‚0.9545963993528858 0loss/SRqqt2/detection_loss‚0.717549280237822 %loss/total_loss‚8.409816788642448´Ç\z“è‚(:ˆ ÝçÒãÄ¡¥ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SªZ"ùé‚(:î ÝçÒâ¥ù¨" disk.dm-0.out‚28338.2958984375 disk.nvme0n1.in‚ 290.265625' disk.nvme0c0n1.out‚28338.2958984375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 204.1171875 disk.dm-1.in‚0 disk.dm-1.out‚ 204.1171875 disk.rbd3.out‚0* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 network.recv‚ 1.547806e+06$ disk./.usageGB‚545.6426315307617 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 290.265625 disk.sda.out‚ 204.1171875 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875$ memory_percent‚95.47901153564453# proc.memory.rssMB‚ 2817.87109375 cpu‚26.018305860621226 disk.dm-0.in‚ 290.265625 disk.sda3.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚28338.2958984375 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚97.85693359375% proc.memory.availableMB‚ 740.71875 disk.sda1.out‚0 disk.drbd2165.in‚0 network.sent‚ 1.0919635e+07( disk./.usagePercent‚36.6185348266915ñu<€“ê‚(:ˆ ÝçÒöòô¤ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.411' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8822 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º{éþë‚(:ó ÝçÒ©¸§ disk.drbd2165.out‚0 disk.sda.out‚ 205.9765625 cpu‚26.017469637039152 network.sent‚ 1.0969651e+07$ disk./.usageGB‚545.6459197998047 disk.dm-1.out‚ 205.9765625 disk.sda1.in‚0( disk.nvme0c0n1.out‚28363.92822265625$ memory_percent‚95.48017978668213( proc.memory.availableMB‚ 740.52734375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 291.09375 disk.sda2.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.618755505186016 disk.rbd1.in‚ 0.01171875 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.out‚ 205.9765625 disk.nvme0c0n1.in‚ 291.09375 disk.sda3.in‚0* proc.memory.percent‚17.198920249938965 network.recv‚ 1.552228e+06# disk.dm-0.out‚28363.92822265625 disk.nvme0n1.in‚ 291.09375& disk.nvme0n1.out‚28363.92822265625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚97.85693359375 disk.rbd3.out‚0# proc.memory.rssMB‚ 2817.87109375 ¸y%2ì- _wandbruntime‚5878  _runtime‚5878T¬ÄþÉíà ' lr-AdamW/pg2‚0.00035562569844423895 trainer/global_step‚6899 % _timestamp‚1.7842786860718868e+09 _runtime‚5878.197487384 _step‚565 ' lr-AdamW/pg1‚0.00035562569844423895§}OqÎîÈ _step‚565 ' lr-AdamW/pg1‚0.00035562569844423895 ' lr-AdamW/pg2‚0.00035562569844423895 trainer/global_step‚6899 % _timestamp‚1.7842786860718868e+09 _runtime‚5878.197487384µyÏào2ï- _wandbruntime‚5878  _runtime‚5878÷ö0ô'ðîO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842786860729704e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26954157 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7286901162854 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1056099954594816 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1056099596216320 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1056099596216320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7298034755584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚5878.202395793 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63916677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7298376094720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7287241712744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376•_¹žñóO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842786860729704e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26954157 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7286901162854 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1056099954594816 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1056099596216320 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1056099596216320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7298034755584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚5878.202395793 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63916677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7298376094720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7287241712744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¶Vìs›2ò- _wandbruntime‚5879  _runtime‚5879 RÝ¡ý&ó÷M LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1048948802490368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1050840600221696 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54723616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26957974 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1043552705044480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7299047382528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7299388728832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚5879.028194788 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1056248208258560 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118649359 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1048948819529728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54723614 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚567 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26957705 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12090963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14867008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231086328563712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1050840957811058 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1043552688005120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7299388728832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14866742 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118639366 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63915752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63925743 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1048948819529728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118649359 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002“f''ôüM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1048948802490368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1050840600221696 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54723616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26957974 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1043552705044480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7299047382528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7299388728832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚5879.028194788 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1056248208258560 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118649359 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1048948819529728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54723614 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚567 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26957705 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12090963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14867008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231086328563712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1050840957811058 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1043552688005120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7299388728832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14866742 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118639366 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63915752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63925743 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1048948819529728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118649359 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002·­-2õ- _wandbruntime‚5879  _runtime‚5879°ˆ+ öƒ _runtime‚ 5879.02958941 _step‚568 1loss/SRqqt2/detection_loss‚0.6302937223251144 epoch‚17 4loss/FBt2/assignment_loss‚0.00044549091263129876 0loss/FRt1/detection_loss‚0.48684470049340073 0loss/SRqqt1/detection_loss‚1.019662573193909 /loss/FBt1/detection_loss‚0.4114240040838386 trainer/global_step‚6899 % _timestamp‚1.7842786869042013e+09 2loss/SRqqt2/assignment_loss‚0.5413314861835943 2loss/FBt1/assignment_loss‚0.010805219365300888 .loss/FBt2/detection_loss‚0.689184729354801 $loss/total_loss‚8.13032302604698 0loss/FRt1/assignment_loss‚1.6465234130039132 0loss/FRt2/assignment_loss‚0.9431782096680712 2loss/SRqqt1/assignment_loss‚0.4192615897299081 /loss/FRt2/detection_loss‚1.3313678877324964]µ–÷ˆ 4loss/FBt2/assignment_loss‚0.00044549091263129876 0loss/FRt1/detection_loss‚0.48684470049340073 0loss/SRqqt1/detection_loss‚1.019662573193909 /loss/FBt1/detection_loss‚0.4114240040838386 trainer/global_step‚6899 % _timestamp‚1.7842786869042013e+09 2loss/SRqqt2/assignment_loss‚0.5413314861835943 2loss/FBt1/assignment_loss‚0.010805219365300888 .loss/FBt2/detection_loss‚0.689184729354801 $loss/total_loss‚8.13032302604698 0loss/FRt1/assignment_loss‚1.6465234130039132 0loss/FRt2/assignment_loss‚0.9431782096680712 2loss/SRqqt1/assignment_loss‚0.4192615897299081 /loss/FRt2/detection_loss‚1.3313678877324964 _runtime‚ 5879.02958941 _step‚568 1loss/SRqqt2/detection_loss‚0.6302937223251144 epoch‚17¸-*Þ†žø‚(:“ ŸÝçҶǣ¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wë Nù‚(:ù ŸÝçÒϼ½§ disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 206.5703125$ disk./.usageGB‚545.6459197998047 disk.rbd1.in‚ 0.01171875 disk.rbd3.out‚0 disk.dm-0.in‚ 291.80859375 disk.rbd0.in‚0 disk.rbd0.out‚0% proc.memory.availableMB‚ 740.78125* proc.memory.percent‚17.198920249938965 disk.rbd1.out‚97.91943359375# disk.dm-0.out‚28370.79833984375 network.recv‚ 1.555077e+06" disk.nvme0c0n1.in‚ 291.80859375 disk.sda3.out‚ 206.5703125# proc.memory.rssMB‚ 2817.87109375 cpu‚26.016707653755´¼Ù¶024 disk.dm-1.in‚0$ memory_percent‚95.47863006591797 proc.cpu.threads‚17& disk.nvme0n1.out‚28370.79833984375 disk.sda1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 291.80859375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 206.5703125 disk.sda3.in‚0 network.sent‚ 1.0986771e+07* disk./.usagePercent‚36.618755505186016 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚28370.79833984375H>–ÿ’ú‚(:‡ ®ÝçÒ㌥ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚244.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.805 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«9<û‚(:ý ®ÝçÒ•ß™§ disk.nvme0n1.in‚ 291.80859375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda.out‚ 206.96484375 disk.sda2.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 291.80859375# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 disk.sda3.out‚ 206.96484375 disk.rbd1.out‚97.91943359375# disk.dm-0.out‚28377.41943359375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 291.80859375' proc.memory.availableMB‚ 740.8046875 cpu‚26.015910522888984 network.sent‚ 1.1038892e+07 disk.sda1.in‚0 network.recv‚ 1.559182e+06$ disk./.usageGB‚545.6459007263184 disk.dm-1.out‚ 206.96484375& disk.nvme0n1.out‚28377.41943359375 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚95.47848701477051) disk./.usagePercent‚36.61875422514835 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875( disk.nvme0c0n1.out‚28377.41943359375Óÿ ÷“ü‚(:ˆ ½ÝçÒð€‚¥ gpu.0.gpu‚92.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†:¿¡ðý‚(:å ½ÝçÒ•åǧ network.sent‚ 1.1044002e+07* disk./.usagePercent‚36.618755505186016$ disk./.usageGB‚545.6459197998047 disk.nvme0n1.in‚ 292.63671875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 292.63671875# proc.memory.rssMB‚ 2817.87109375 cpu‚26.01511157275408 disk.dm-0.in‚ 292.63671875 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚28384.568359375 disk.sda.out‚207.25 disk.sda3.out‚207.25$ memory_percent‚95.48001289367676' proc.memory.availableMB‚ 740.5546875 network.recv‚ 1.562862e+06* proc.memory.percent‚17.198920249938965 disk.dm-1.out‚207.25 disk.sda1.in‚0 disk.rbd1.out‚97.98193359375 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0$ disk.nvme0n1.out‚28384.568359375! disk.dm-0.out‚28384.568359375 disk.sda1.out‚0 disk.sda2.in‚0 ŠÎ2þ- _wandbruntime‚5918  _runtime‚5918ïCDIÅÿ¿ & lr-AdamW/pg1‚0.0003539868306433267 & lr-AdamW/pg2‚0.0003539868306433267 trainer/global_step‚6949 % _timestamp‚1.7842787268028193e+09 _runtime‚ 5918.9294095 _step‚569á`ÚýÊ€Ä _step‚569 & lr-AdamW/pg1‚0.0003539868306433267 & lr-AdamW/pg2‚0.0003539868306433267 trainer/global_step‚6949 % _timestamp‚1.7842787268028193e+09 _runtime‚ 5918.9294095¹Ã1Ž2- _wandbruntime‚5918  _runtime‚5918§¿tvô'‚îO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55097584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1063512343199744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7348730257408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7430386760704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27143766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64371129 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14967569 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232724337731584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64361152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27144040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55097582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12176468 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55097584 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7349071596544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119468713 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1058069228576610 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1050731727966208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7337858199652 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1056163629981696 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1056163612942336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240154743104512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12176471 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7842787268035898e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7348730257408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7337517649762 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64371129 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7430298183680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1063512701578240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232724356343808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7349071596544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240154635915264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119458734 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5918.929479131 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1056163629981696 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1063512343199744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14967298 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169{Àèù'ƒóO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14967569 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232724337731584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64361152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27144040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55097582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12176468 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55097584 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7349071596544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119468713 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1058069228576610 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1050731727966208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7337858199652 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1056163629981696 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1056163612942336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240154743104512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12176471 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7842787268035898e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7348730257408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7337517649762 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64371129 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7430298183680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1063512701578240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232724356343808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7349071596544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240154635915264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119458734 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5918.929479131 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1056163629981696 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1063512343199744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14967298 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55097584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1063512343199744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7348730257408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7430386760704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27143766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64371129 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541ºMQúC2„- _wandbruntime‚5919  _runtime‚5919wKÀ …øM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55105214 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7431318057984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5919.550908491 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12178216 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1063660955241984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1056310853971968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7350084230656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1063660596856320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1056310871011328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12178213 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232757766490112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1058216750302204 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1050878238638080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240189084548096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27147565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1056310853971968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1058217107891566 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119475418 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55105214 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55105216 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7338528703484 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119485411 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14969352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚571 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7350084230656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64380195 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7338869253486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240189193827328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27147842 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7431408724992 NADeviceStatsMonitor.on_train_batch_end/active.large_pooów>l.allocated‚55105216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1050878221598720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14969626 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119485411 trainer/global_step‚6949 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64380195 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64370204 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1056310871011328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119475418 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232757785102336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7349742884352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1063660955241984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 jc™'†ýM _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1056310871011328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12178213 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232757766490112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1058216750302204 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1050878238638080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚240189084548096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27147565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1056310853971968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1058217107891566 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119475418 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55105214 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55105216 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7338528703484 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119485411 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14969352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚571 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7350084230656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64380195 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7338869253486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240189193827328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27147842 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7431408724992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55105216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1050878221598720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14969626 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119485411 trainer/global_step‚6949 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64380195 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64370204 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1056310871011328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119475418 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232757785102336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7349742884352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1063660955241984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55105214 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7431318057984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5919.550908491 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12178216 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1063660955241984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1056310853971968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7350084230656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1063660596856320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0»9&c 2‡- _wandbruntime‚5919  _runtime‚5919`w¸ ˆƒ /loss/FRt1/assignment_loss‚1.622212691109498 0loss/FRt2/assignment_loss‚0.8835362073537466 /loss/FRt2/detection_loss‚1.1822914590527354 1loss/SRqqt2/detection_loss‚0.6800385084746815 .loss/FBt2/detection_loss‚0.864452413706617 % _timestamp‚1.7842787274267554e+09 2loss/SRqqt1/assignment_loss‚0.5668217688154742 2loss/FBt1/assignment_loss‚0.013977199699581418 epoch‚17 trainer/global_step‚6949 _runtime‚5919.552146969 1loss/SRqqt1/detection_loss‚0.9601091130113244 _step‚572 2loss/SRqqt2/assignment_loss‚0.5545463670196977 3loss/FBt2/assignment_loss‚0.0008002737640915504 /loss/FRt1/detection_loss‚0.5622730677301521 /loss/FBt1/detection_loss‚0.4058422003971132 %loss/total_loss‚8.296901270134713È%ˆÕ‰ˆ 2loss/SRqqt2/assignment_loss‚0.5545463670196977 3loss/FBt2/assignment_loss‚0.0008002737640915504 /loss/FRt1/detection_loss‚0.5622730677301521 /loss/FBt1/detection_loss‚0.4058422003971132 %loss/total_loss‚8.296901270134713 /loss/FRt1/assignment_loss‚1.622212691109498 0loss/FRt2/assignment_loss‚0.8835362073537466 /loss/FRt2/detection_loss‚1.1822914590527354 1loss/SRqqt2/detection_loss‚0.6800385084746815 .loss/FBt2/detection_loss‚0.864452413706617 % _timestamp‚1.7842787274267554e+09 2loss/SRqqt1/assignment_loss‚0.5668217688154742 2loss/FBt1/assignment_loss‚0.013977199699581418 epoch‚17 trainer/global_step‚6949 _runtime‚5919.552146969 1loss/SRqqt1/detection_loss‚0.9601091130113244 _step‚572¼(B©Ú“Š‚(:ˆ ÌÝçÒ¸³à¥ gpu.0.gpu‚50.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢%étÿ‹‚(:ô ÌÝç҇΂¨) disk./.usagePercent‚36.61876088134424 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0$ proc.memory.availableMB‚740.5625 disk.sda.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 293.3515625 disk.sda.out‚ 207.3984375 disk.sda1.in‚0 disk.sda2.out‚0# disk.dm-0.out‚28394.06494140625 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 293.3515625$ memory_percent‚95.47996520996094 proc.cpu.threads‚17 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 293.3515625 disk.sda3.out‚ 207.3984375 disk.dm-1.out‚ 207.3984375* proc.memory.percent‚17.198920249938965 network.recv‚ 1.565888e+06( disk.nvme0c0n1.out‚28394.06494140625 disk.dm-1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2817.87109375$ disk./.usageGB‚545.6459999084473 disk.rbd0.out‚0 disk.rbd1.out‚97.98193359375 cpu‚26.014315806366888& disk.nvme0n1.out‚28394.06494140625 network.sent‚ 1.1060007e+07þ´בŒ‚(:† ÛÝçÒõŸÁ¤ gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ&&«‚(:ù ÛÝçÒªàݦ disk.drbd2165.out‚0# disk./.usageGB‚545.646240234375 disk.sda2.out‚0 disk.dm-0.in‚ 294.1796875& disk.nvme0n1.out‚28401.70361328125 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 294.1796875* proc.memory.percent‚17.198920249938965 cpu‚26.013539808264806 disk.sda3.in‚0 disk.rbd1.out‚98.04443359375$ memory_percent‚95.47960758209229 disk.rbd1.in‚ 0.01171875# disk.dm-0.out‚28401.70361328125( proc.memory.availableMB‚ 740.62109375 proc.cpu.threads‚17 disk.dm-1.out‚ 207.74609375 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd3.out‚0 network.sent‚ 1.1112093e+07 disk.sda3.out‚ 207.74609375 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚28401.70361328125 network.recv‚ 1.57064e+06 disk.rbd3.in‚0# proc.memory.rssMB‚ 2817.87109375) disk./.usagePercent‚36.61877700981889 disk.sda2.in‚0 disk.nvme0n1.in‚ 294.1796875 disk.drbd2165.in‚0 disk.sda.out‚ 207.74609375 disk.sda1.out‚0«¹"˜“Ž‚(:ˆ êÝçÒÆÂæ¤ gpu.0.gpu‚70.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¿…ü‚(:ñ êÝçÒ›¥÷¦ cpu‚26.012795832564436 disk.nvme0n1.in‚ 294.89453125 disk.rbd1.out‚98.04443359375# proc.memory.rssMB‚ 2817.87109375$ memory_percent‚95.48094272613525 disk.dm-1.out‚207.9375 disk.rbd3.in‚0 network.recv‚ 1.573833e+06# disk.dm-0.out‚28411.30126953125" disk.nvme0c0n1.in‚ 294.89453125 disk.sda.out‚207.9375 disk.sda3.out‚207.9375) disk./.usagePercent‚36.61878187396204 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚28411.30126953125 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875( proc.memory.availableMB‚ 740.40234375# disk./.usageGB‚545.646312713623 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 disk.rbd3.out‚0 network.sent‚ 1.1115015e+07 disk.dm-0.in‚ 294.89453125 disk.drbd2165.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚28411.30126953125ú#ý2- _wandbruntime‚5959  _runtime‚5959˜ç¬¼Ç‘Á & lr-AdamW/pg1‚0.0003523404062278245 & lr-AdamW/pg2‚0.0003523404062278245 trainer/global_step‚6999 % _timestamp‚1.7842787675420332e+09 _runtime‚5959.667666403 _step‚573,ÈÌmÌ’Æ % _timestamp‚1.7842787675420332e+09 _runtime‚5959.667666403 _step‚573 & lr-AdamW/pg1‚0.0003523404062278245 & lr-AdamW/pg2‚0.0003523404062278245 trainer/global_step‚6999½tº·2“- _wandbruntime‚5959  _runtime‚5959@•ò'”ìO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64825581 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1070925448561664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120304765 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12263721 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚574 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7399425759232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784278767543317e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64825581 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55479184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234395775657984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55479182 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15071100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 5959.66907351 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1063525681463296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063525681463296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1070925090183168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1058057261559808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6999 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7388134136670 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7481418214912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1065445736246368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7399767098368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7399767098368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1065445378657118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120294786 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64815604 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1070925090183168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0üÌ4`•ñO WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1070925448561664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7481418214912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1065445736246368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7399767098368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7399767098368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1065445378657118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120294786 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64815604 S>DeviceStatsMonitor.on_train_batch_start/active_byte8p”—s.all.freed‚1070925090183168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64825581 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1070925448561664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120304765 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12263721 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚574 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7399425759232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784278767543317e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64825581 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55479184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234395775657984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55479182 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15071100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 5959.66907351 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1063525681463296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063525681463296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1070925090183168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1058057261559808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6999 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7388134136670 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7400438386176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚5960.285120875 _step‚575 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389485740394 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64824656 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1071073702225408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55486814 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55486816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120321463 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234429204416512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1063672905453568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27338568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7400779732480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55486816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64824656 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1065593257972074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55486814 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64834647 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7482529120768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063672922492928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12265463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15073378 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚6999 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1071073702225408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0$õêà'˜üM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7482442648064 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7400438386176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚5960.285120875 _step‚575 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389485740394 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64824656 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1071073702225408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55486814 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55486816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120321463 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234429204416512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1063672905453568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27338568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7400779732480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55486816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64824656 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1065593257972074 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55486814 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64834647 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7482529120768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063672922492928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12265463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15073378 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚6999 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1071073702225408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15073105 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1058203772231680 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1058203755192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120321463 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7400779732480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241911752149504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241911647064576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0¿}<Ï2™- _wandbruntime‚5960  _runtime‚5960ìœ'Úˆ 0loss/FRt1/assignment_loss‚1.6888238486503906 0loss/FRt2/assignment_loss‚0.9259658976189779 3loss/SRqqt1/assignment_loss‚0.44730065699006544 /loss/FRt1/detection_loss‚0.5424354081222981 /loss/FBt2/detection_loss‚1.1353909669281996 %loss/total_loss‚8.425203431286592 _runtime‚5960.286616038 3loss/FBt1/assignment_loss‚0.0068390401292576924 4loss/FBt2/assignment_loss‚0.00033224158068170925 /loss/FRt2/detection_loss‚1.2967705127496976 trainer/global_step‚6999 % _timestamp‚1.7842787681612954e+09 1loss/SRqqt1/detection_loss‚0.7979203325073073 0loss/SRqqt2/detection_loss‚0.692044790323975 0loss/FBt1/detection_loss‚0.44021785293875454 _step‚576 2loss/SRqqt2/assignment_loss‚0.4511618827469876 epoch‚17d òœ› %loss/total_loss‚8.425203431286592 _runtime‚5960.286616038 3loss/FBt1/assignment_loss‚0.0068390401292576924 4loss/FBt2/assignment_loss‚0.00033224158068170925 /loss/FRt2/detection_loss‚1.2967705127496976 trainer/global_step‚6999 % _timestamp‚1.7842787681612954e+09 1loss/SRqqt1/detection_loss‚0.7979203325073073 0loss/SRqqt2/detection_loss‚0.692044790323975 0loss/FBt1/detection_loss‚0.44021785293875454 _step‚576 2loss/SRqqt2/assignment_loss‚0.4511618827469876 epoch‚17 0loss/FRt1/assignment_loss‚1.6888238486503906 0loss/FRt2/assignment_loss‚0.9259658976189779 3loss/SRqqt1/assignment_loss‚0.44730065699006544 /loss/FRt1/detection_loss‚0.5424354081222981 /loss/FBt2/detection_loss‚1.1353909669281996Àíf$)“œ‚(:ˆ ùÝçÒί°¥ gpu.0.gpu‚87.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OåC‚(:ù ùÝçÒ˨ѧ% disk.nvme0n1.out‚28414.0302734375 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 cpu‚26.011967891915454 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 1.1133102e+07" disk.nvme0c0n1.in‚ 294.89453125$ memory_percent‚95.47955989837646) disk./.usagePercent‚36.61878725012026 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 208.23828125 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚98.10693359375 disk.dm-0.in‚ 294.89453125 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚28414.0302734375 disk.nvme0n1.in‚ 294.89453125$ disk./.usageGB‚545.6463928222656 disk.sda1.out‚0' disk.nvme0c0n1.out‚28414.0302734375( proc.memory.availableMB‚ 740.62890625 network.recv‚ 1.5773e+06 disk.rbd0.in‚0 disk.sda.out‚ 208.23828125 disk.sda3.out‚ 208.23828125# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965»¥øó“ž‚(:ˆ ˆÞçҸ gpu.0.gpu‚49.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„²6­Ÿ‚(:÷ ˆÞçÒðＧ disk.nvme0n1.in‚ 295.72265625* proc.memory.percent‚17.198920249938965 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 network.sent‚ 1.1185468e+07 network.recv‚ 1.58151e+06$ disk./.usageGB‚545.6459503173828 disk.rbd1.in‚ 0.01171875$ memory_percent‚95.47967910766602& proc.memory.availableMB‚ 740.609375 cpu‚26.01127621128037 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2817.87109375 disk.drbd2165.in‚0 disk.dm-1.out‚ 208.3515625# disk.dm-0.out‚28421.79443359375& disk.nvme0n1.out‚28421.79443359375 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda.out‚ 208.3515625 disk.rbd1.out‚98.10693359375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 295.72265625( disk.nvme0c0n1.out‚28421.79443359375 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0) disk./.usagePercent‚36.61875755324629 disk.sda2.in‚0 disk.sda3.out‚ 208.3515625 disk.dm-0.in‚ 295.72265625œ®=n‘ ‚(:† —ÞçÒÎÔÏ¢ gpu.0.gpu‚80.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚252.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦FçAò¡‚(:ç —ÞçÒÚ­«¦* disk./.usagePercent‚36.618838195619574 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚26.01047290038251 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 208.71875 disk.dm-0.in‚296.4375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚28428.06982421875 disk.sda2.in‚0 disk.dm-1.out‚ 208.71875 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1190513e+07 network.recv‚ 1.585165e+06 disk.sda3.in‚0& proc.memory.availableMB‚ 494.359375 disk.nvme0n1.in‚296.4375& disk.nvme0n1.out‚28428.06982421875 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.198920249938965 disk.sda3.out‚ 208.71875 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2817.87109375 disk.rbd3.out‚0$ memory_percent‚96.98266983032227$ disk./.usageGB‚545.6471519470215 disk.nvme0c0n1.in‚296.4375 disk.sda2.out‚0 disk.rbd1.out‚98.23193359375# disk.dm-0.out‚28428.06982421875 þ2¢- _wandbruntime‚6000  _runtime‚6000ÙÁæ”Å£¿ % lr-AdamW/pg1‚0.000350686530944496 % lr-AdamW/pg2‚0.000350686530944496 trainer/global_step‚7049 % _timestamp‚1.7842788081283767e+09 _runtime‚6000.253991009 _step‚577&{˽Ê¤Ä trainer/global_step‚7049 % _timestamp‚1.7842788081283767e+09 _runtime‚6000.253991009 _step‚577 % lr-AdamW/pg1‚0.000350686530944496 % lr-AdamW/pg2‚0.000350686530944496ÁöÈçä2¥- _wandbruntime‚6000  _runtime‚6000lI>YŦîO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1065382795153408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55860782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243599755552768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121140817 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65280033 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1078338195545088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7450121261056 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7450121261056 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1070887732944896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55860782 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚7049 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27525486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12350968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7438750623578 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65280033 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243599866936320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1078337837166592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236067213584384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12350971 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55860784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1065382795153408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55860782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243599755552768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121140817 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65280033 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1078338195545088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7450121261056 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7450121261056 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1070887732944896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55860782 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚7049 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27525486 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12350968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7438750623578 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65280033 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243599866936320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1078337837166592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236067213584384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12350971 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55860784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169Ât<«š2¨- _wandbruntime‚6000  _runtime‚6000!Ášãþ&©øM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121147522 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚7049 _runtime‚6000.876847864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121147522 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1071034973974528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15176532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1065529305825280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1065529288785920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7440102227302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1078486090823168 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121157515 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55868416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1071034956935168 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1071034956935168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1071034973974528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7533659326976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7451133888000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55868414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27528997 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121157515 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1078486449208832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7533570757120 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1072969050463220 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7439761677300 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15176284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872&.Õp… ªýM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243634320282112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15176532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1065529305825280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1065529288785920 \JDeviceStatsMonit×> D~or.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7440102227302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1078486090823168 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121157515 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55868416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1071034956935168 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1071034956935168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1071034973974528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7533659326976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7451133888000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55868414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27528997 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121157515 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1078486449208832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7533570757120 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1072969050463220 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7439761677300 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15176284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121147522 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚7049 _runtime‚6000.876847864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121147522 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1071034973974528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65724508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27716133 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15277912 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65724508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1078249784426496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7489367110486 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1085750942528512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1078249767387136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1080197678818134 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12438218 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080198036407384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7489707660376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12438221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237738651510784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65734485 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121976869 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56242384 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1072708328747008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7500816762880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237738670123008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7501158102016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7500816762880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245322473918464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚582 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65734485 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7583803795456 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1078249784426496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1085750584150016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÚrðcÂ'·óO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1085750584150016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚6040.999519599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56242384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65724508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27716133 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15277912 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65724508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1078249784426496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7489367110486 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1085750942528512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1078249767387136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1080197678818134 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12438218 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080198036407384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7489707660376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12438221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237738651510784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65734485 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121976869 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56242384 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1072708328747008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7500816762880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237738670123008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7501158102016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7500816762880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245322473918464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚582 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65734485 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7583803795456 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1078249784426496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsºÅU€7Monitor.on_train_batch_start/segment.all.freed‚0Æf*áf2¸- _wandbruntime‚6041  _runtime‚6041è„Tþ&¹øM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27719912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121983574 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _step‚583 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65743551 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1085899196192256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7501829389824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121993567 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56250016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15279691 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27719654 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65733560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245356923495936 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1072854822379520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65733560 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1078397025456128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7501829389824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245356810022400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12439963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15279946 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237772080269312 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121993567 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1085899196192256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976+›Á='ºýM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121993567 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1085899196192256 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27719912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121983574 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _step‚583 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65743551 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1085899196192256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7501829389824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121993567 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56250016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15279691 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27719654 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65733560 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245356923495936 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1072854822379520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65733560 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1078397025456128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7501829389824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245356810022400 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12439963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15279946 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237772080269312 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169Ç]x,2»- _wandbruntime‚6041  _runtime‚6041ÙI7¼‚ 2loss/SRqqt1/assignment_loss‚0.4573160932677926 2loss/FBt1/assignment_loss‚0.012038541233843255 2loss/FBt2/assignment_loss‚0.026411131788720992 /loss/FRt1/detection_loss‚0.5514525187770675 trainer/global_step‚7099 _step‚584 /loss/FRt2/detection_loss‚1.2379647907045195 1loss/SRqqt2/detection_loss‚0.6756365199331665 .loss/FBt1/detection_loss‚0.506223056332431 /loss/FBt2/detection_loss‚0.7904448764255414 _runtime‚6041.620232974 0loss/FRt2/assignment_loss‚0.8953039562198478 2loss/SRqqt2/assignment_loss‚0.5188288352295132 %loss/total_loss‚8.267108155418686 1loss/SRqqt1/detection_loss‚0.9320333981433487 epoch‚17 $ _timestamp‚1.784278849494822e+09 0loss/FRt1/assignment_loss‚1.6634544373628934š ½‡ %loss/total_loss‚8.267108155418686 1loss/SRqqt1/detection_loss‚0.9320333981433487 epoch‚17 $ _timestamp‚1.784278849494822e+09 0loss/FRt1/assignment_loss‚1.6634544373628934 2loss/SRqqt1/assignment_loss‚0.4573160932677926 2loss/FBt1/assignment_loss‚0.012038541233843255 2loss/FBt2/assignment_loss‚0.026411131788720992 /loss/FRt1/detection_loss‚0.5514525187770675 trainer/global_step‚7099 _step‚584 /loss/FRt2/detection_loss‚1.2379647907045195 1loss/SRqqt2/detection_loss‚0.6756365199331665 .loss/FBt1/detection_loss‚0.506223056332431 /loss/FBt2/detection_loss‚0.7904448764255414 _runtime‚6041.620232974 0loss/FRt2/assignment_loss‚0.8953039562198478 2loss/SRqqt2/assignment_loss‚0.5188288352295132È3 û8’¾‚(:‡ ÄÞçÒ¸£œ¥ gpu.0.gpu‚86.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.973 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÓC¿‚(:û ÄÞçÒºíɧ& disk.nvme0n1.out‚28459.00634765625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 297.98046875 disk.dm-0.in‚ 297.98046875 cpu‚26.00829098164787$ disk./.usageGB‚545.6450271606445 disk.sda3.in‚0* proc.memory.percent‚17.198920249938965 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚98.29443359375 disk.rbd3.in‚0# disk.dm-0.out‚28459.00634765625 network.recv‚ 1.596739e+06 disk.nvme0n1.in‚ 297.98046875 disk.sda.out‚ 211.5390625 disk.dm-1.out‚ 211.5390625( proc.memory.availableMB‚ 494.23828125 network.sent‚ 1.1261588e+07 disk.sda.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2817.87109375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚28459.00634765625 disk.sda2.out‚0 disk.sda3.out‚ 211.5390625* disk./.usagePercent‚36.618695599423006 disk.rbd1.in‚ 0.01171875$ memory_percent‚96.98340892791748 proc.cpu.threads‚17ý§Ì!“À‚(:ˆ ÓÞçÒÃÈä gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãHK8Á‚(:ý ÓÞçÒ–—Ц network.sent‚ 1.1324605e+07$ disk./.usageGB‚545.6461410522461 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 214.14453125 network.recv‚ 1.60122e+06 disk.nvme0n1.in‚ 298.80859375 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 298.80859375 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚98.35693359375 disk.rbd3.out‚0# disk.dm-0.out‚28475.94091796875' proc.memory.availableMB‚ 494.5546875* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚28475.94091796875 disk.sda3.out‚ 214.14453125 disk.sda1.in‚0& disk.nvme0n1.out‚28475.94091796875 disk.sda.in‚0* disk./.usagePercent‚36.618770353623006 disk.rbd3.in‚0 cpu‚26.007623925206268 disk.dm-0.in‚ 298.80859375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 214.14453125$ memory_percent‚96.98147773742676# proc.memory.rssMB‚ 2817.87109375º¤VHžÂ‚(:“ âÞçÒŃì¤ gpu.0.gpu‚88.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚258.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.717200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ïcMÂ(:ú âÞçÒ´“§$ disk./.usageGB‚545.6449012756348 disk.sda3.in‚0) disk./.usagePercent‚36.61868715117437 disk.sda.out‚ 215.94921875 disk.sda2.in‚0 disk.sda3.out‚ 215.94921875 disk.dm-0.in‚ 299.5234375# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 cpu‚26.00682456964757 network.recv‚ 1.603859e+06 disk.dm-1.out‚ 215.94921875 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚28493.03369140625& disk.nvme0n1.out‚28493.03369140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.98145389556885 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 299.5234375 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚28493.03369140625 proc.cpu.threads‚17 disk.rbd1.out‚98.35693359375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 299.5234375 disk.rbd0.in‚0( proc.memory.availableMB‚ 494.55859375 network.sent‚ 1.1331571e+07Ǫ–q2Ä- _wandbruntime‚6081  _runtime‚6081õß2ÇÅÁ _step‚585 & lr-AdamW/pg1‚0.0003473568531473685 & lr-AdamW/pg2‚0.0003473568531473685 trainer/global_step‚7149 % _timestamp‚1.7842788896399832e+09 _runtime‚6081.765587149W­‚cÌÆÆ & lr-AdamW/pg1‚0.0003473568531473685 & lr-AdamW/pg2‚0.0003473568531473685 trainer/global_step‚7149 % _timestamp‚1.7842788896399832e+09 _runtime‚6081.765587149 _step‚585ɳÝ;2Ç- _wandbruntime‚6081  _runtime‚6081ùܺÈîO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239410108049408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15380520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚586 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7539983597394 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7842788896410007e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1085611818868736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56623984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122812921 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚7149 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66188937 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66188937 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56623984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27906245 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239410089437184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1080033845301248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7540324147284 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122802942 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66178960 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1093163331133440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1093163331133440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7551853603840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7551512264704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1093163689511936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7634848486912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1087573828898642 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1080033862340608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66178960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247045043016192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56623982 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1087574186487892 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1085611818868736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1093163689511936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12525468 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1085611835908096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7551512264704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591%`aù'ÉóO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56623984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122812921 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚7149 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66188937 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66188937 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56623984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27906245 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239410089437184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1080033845301248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7540324147284 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122802942 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66178960 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1093163331133440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1093163331133440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7551853603840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7551512264704 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1093163689511936 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7634848486912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1087573828898642 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1080033862340608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66178960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247045043016192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56623982 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1087574186487892 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1085611818868736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1093163689511936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12525468 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1085611835908096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7551512264704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239410108049408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15380520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚586 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7539983597394 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7842788896410007e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1085611818868736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ÊA5ãs2Ê- _wandbruntime‚6082  _runtime‚60824¹êþ&ËøM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7635956015104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247079381446656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7635863250944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7540994651116 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122829619 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7552524891648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842788902603683e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66188012 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56631616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66188012 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27909798 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1093311943175680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7552866237952 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27910083 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122829619 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1093311943175680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15382585 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚7149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66198003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1087721350624236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56631614 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1085759076937728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1087721708213598 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12527213 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1085759059898368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247079492823040;EÒSÄ ÌýM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122829619 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7552524891648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842788902603683e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66188012 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56631616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66188012 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27909798 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1093311943175680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7552866237952 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27910083 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122829619 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1093311943175680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15382585 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚7149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66198003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1087721350624236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56631614 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1085759076937728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1087721708213598 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12527213 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1085759059898368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247079492823040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7635956015104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247079381446656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7635863250944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7540994651116˺ô‰i2Í- _wandbruntime‚6082  _runtime‚6082SJ[€ ΃ /loss/FRt2/assignment_loss‚0.891935136408176 .loss/FRt2/detection_loss‚1.330239557257177 _step‚588 2loss/SRqqt1/assignment_loss‚0.5179039032032745 3loss/FBt2/assignment_loss‚5.323881541379194e-05 1loss/SRqqt2/detection_loss‚0.6915095878645586 .loss/FBt2/detection_loss‚0.648993106931448 trainer/global_step‚7149 _runtime‚6082.387511385 /loss/FRt1/assignment_loss‚1.679113716373506 2loss/SRqqt2/assignment_loss‚0.5092735917549075 /loss/FRt1/detection_loss‚0.5535145427884216 %loss/total_loss‚8.261048743053866 epoch‚17 % _timestamp‚1.7842788902621737e+09 3loss/FBt1/assignment_loss‚0.0022522179890975013 1loss/SRqqt1/detection_loss‚1.0225832370875434 0loss/FBt1/detection_loss‚0.41367690658034195Z¸%ψ 2loss/SRqqt1/assignment_loss‚0.5179039032032745 3loss/FBt2/assignment_loss‚5.323881541379194e-05 1loss/SRqqt2/detection_loss‚0.6915095878645586 .loss/FBt2/detection_loss‚0.648993106931448 trainer/global_step‚7149 _runtime‚6082.387511385 /loss/FRt1/assignment_loss‚1.679113716373506 2loss/SRqqt2/assignment_loss‚0.5092735917549075 /loss/FRt1/detection_loss‚0.5535145427884216 %loss/total_loss‚8.261048743053866 epoch‚17 % _timestamp‚1.7842788902621737e+09 3loss/FBt1/assignment_loss‚0.0022522179890975013 1loss/SRqqt1/detection_loss‚1.0225832370875434 0loss/FBt1/detection_loss‚0.41367690658034195 /loss/FRt2/assignment_loss‚0.891935136408176 .loss/FRt2/detection_loss‚1.330239557257177 _step‚588Ì4Üùi“Ђ(:ˆ ñÞçÒ¸¶ó¢ gpu.0.gpu‚59.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚260.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³öÑ‚(:ö ñÞçÒË´•­) disk./.usagePercent‚36.61868971124972 disk.drbd2165.in‚0 disk.dm-1.out‚ 217.67578125 disk.rbd1.out‚98.41943359375 network.sent‚ 1.1333747e+07! disk.nvme0c0n1.in‚ 300.3515625 disk.sda2.in‚0 cpu‚26.006150262491804 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚28506.2958984375 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875$ disk./.usageGB‚545.6449394226074 disk.rbd3.out‚0 disk.nvme0n1.in‚ 300.3515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚28506.2958984375 disk.sda3.out‚ 217.67578125' proc.memory.availableMB‚ 494.0703125 network.recv‚ 1.60709e+06% disk.nvme0n1.out‚28506.2958984375 disk.sda.in‚0# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 disk.sda.out‚ 217.67578125 disk.dm-0.in‚ 300.3515625$ memory_percent‚96.98443412780762½‚¼‘Ò‚(:† €ßçÒüéؤ gpu.0.gpu‚84.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ÊšÓ‚(:÷ €ßçÒÍ–ý¦* disk./.usagePercent‚36.618689967257254( disk.nvme0c0n1.out‚28513.91650390625 disk.sda2.in‚0& disk.nvme0n1.out‚28513.91650390625 disk.drbd2165.in‚0 network.sent‚ 1.1398864e+07 disk.sda3.in‚0 disk.dm-1.out‚ 219.2265625 disk.rbd1.out‚98.41943359375 disk.rbd3.in‚0 disk.dm-0.in‚ 300.3515625 disk.nvme0n1.in‚ 300.3515625 disk.rbd0.out‚0 disk.sda3.out‚ 219.2265625$ disk./.usageGB‚545.6449432373047 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 493.890625$ memory_percent‚96.98553085327148 cpu‚26.005370158592356! disk.nvme0c0n1.in‚ 300.3515625 disk.sda.out‚ 219.2265625# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 network.recv‚ 1.611274e+06 disk.sda2.out‚0# disk.dm-0.out‚28513.91650390625 disk.rbd0.in‚0¹ö²“Ô‚(:ˆ ßçÒ§Žè¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å.Õ‚(:ý ßçÒÄ# proc.memory.rssMB‚ 2817.87109375 disk.nvme0n1.in‚ 301.06640625 disk.dm-0.in‚ 301.06640625( proc.memory.availableMB‚ 493.89453125& disk.nvme0n1.out‚28520.47900390625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 219.81640625 disk.rbd3.out‚0# disk.dm-0.out‚28520.47900390625 disk.rbd0.out‚0 network.sent‚ 1.1401929e+07* disk./.usagePercent‚36.618694575392865$ disk./.usageGB‚545.6450119018555 disk.sda1.in‚0 disk.rbd1.in‚ 0.01171875* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚ 219.81640625 disk.sda2.out‚0 network.recv‚ 1.61498e+06 disk.sda2.in‚0 disk.sda3.out‚ 219.81640625 disk.rbd1.out‚98.48193359375 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 301.06640625( disk.nvme0c0n1.out‚28520.47900390625 disk.sda.in‚0 disk.sda3.in‚0 cpu‚26.00470662227944$ memory_percent‚96.98550701141357ÛÜ{72Ö- _wandbruntime‚6122  _runtime‚61227q£†Æ×À & lr-AdamW/pg1‚0.0003456812644925524 & lr-AdamW/pg2‚0.0003456812644925524 trainer/global_step‚7199 % _timestamp‚1.7842789301800723e+09 _runtime‚ 6122.30566748 _step‚589 YEWËØÅ & lr-AdamW/pg1‚0.0003456812644925524 & lr-AdamW/pg2‚0.0003456812644925524 trainer/global_step‚7199 % _timestamp‚1.7842789301800723e+09 _runtime‚ 6122.30566748 _step‚589Í>pöŒ2Ù- _wandbruntime‚6122  _runtime‚6122“º‚´ô'ÚîO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7590600084302 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57005584 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1100576078116864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57005582 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66633412 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1100576078116864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7590940634192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123638994 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57005582 _runtime‚6122.306911608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28097306 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7602207766528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7686248429568 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123648973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248767693507584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28097032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1092973870350336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7602549105664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15484314 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57005584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1094950336568400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1100576436495360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1100576436495360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66643389 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1094949978979150 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1092973887389696 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66643389 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7602207766528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123648973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248767693507584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28097032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1092973870350336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7602549105664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15484314 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57005584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1094950336568400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1100576436495360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1100576436495360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66643389 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1094949978979150 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1092973887389696 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66643389 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7602207766528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1100576078116864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57005582 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888΢é]S2Ü- _wandbruntime‚6122  _runtime‚6122‰bA$Ý÷M K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1093121111379968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1095097500704744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57013214 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7603561739776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123665671 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57013216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6122.927408237 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7603561739776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12614466 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66652455 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1095097858294106 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7591951688026 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12614463 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486635 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248802143567360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7687273917952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 $ _timestamp‚1.784278930801492e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241114956122112 J=DeviceStatsMonitor.oãÈÙn_train_batch_end/active.small_pool.freed‚66642464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28101101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7603220393472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66652455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚7199 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1093121128419328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241114974734336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1100724690159104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15486377 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1100724690159104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57013214 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7591611138024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123665671 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1100724331773440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123655678 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28100840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7687187445248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248802248652288 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1087505889566720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983œ}?,'ÞüM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123655678 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57013216 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1087505906606080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1100724690159104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57013214 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7591611138024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123665671 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1100724331773440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123655678 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28100840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7687187445248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248802248652288 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1087505889566720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1093121111379968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1095097500704744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57013214 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7603561739776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123665671 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57013216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6122.927408237 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7603561739776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12614466 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66652455 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1095097858294106 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7591951688026 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12614463 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486635 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248802143567360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7687273917952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 $ _timestamp‚1.784278930801492e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241114956122112 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66642464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28101101 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7603220393472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66652455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚7199 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1093121128419328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241114974734336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1100724690159104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15486377 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3379344344139099 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7620846033096313 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7251322865486145 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12103673070669174 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.32776179909706116 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6960147619247437 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24024946987628937 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6560530662536621 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5244975090026855 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.790650486946106 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8698999285697937 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.002801120514050126 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833432674408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962183833122253 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969032764434814 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7034876346588135 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4803662896156311 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7411195039749146 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6663458943367004 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7754723429679871 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8657407164573669 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7627173066139221 +particle/specificity‚0.9076398015022278 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39954742789268494 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16092664003372192 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7906768321990967 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14558538794517517 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6785169243812561 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7266062498092651 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.747317373752594 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7241339087486267 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_3‚0.6001158952713013 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961509704589844 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6643376350402832 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.263439416885376 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48003238439559937 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9921030402183533 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5562500357627869 particle/accuracy_1_of_1‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3396086096763611 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7427458763122559 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9970238208770752 % _timestamp‚1.7842789483034515e+09 )loss/val_total_loss‚8.158145491245095 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891303777694702 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6204812526702881 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992997050285339 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3396086096763611 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6551010608673096 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992997050285339 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.348013311624527 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_4‚0.9646546244621277 particle/accuracy_2_of_2‚1 _step‚593 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9165229797363281 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0634046420454979 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2673523426055908 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23579566180706024 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2553541958332062 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1525000035762787 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/sensitivity‚0.6210826635360718 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6747778654098511 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 trainer/global_step‚7199 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7687480449676514 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7918264269828796 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3387208580970764 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994879364967346 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6295425295829773 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6201350092887878 jet/accuracy_2_of_5‚1 epoch‚17 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9964581727981567 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34725406765937805 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5859388113021851 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7147670984268188 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39732661843299866 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02193324826657772 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05475417897105217 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02193324826657772 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6687190532684326 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043982602655887604 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024154186248779297 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7975382208824158 +jet/accuracy_3_of_3‚0.06994573026895523 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02341967076063156 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6625216603279114 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9974327087402344 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5398446917533875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.34770655632019043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6110733151435852 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.714387059211731 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06757380068302155 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49099642038345337 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7821617126464844 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8484181761741638 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5577994585037231 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8484181761741638 particle/accuracy_2_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 (particle/accuracy‚0.8461061120033264 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7510585188865662 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_2_of_4‚0.7945955991744995 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7179927229881287 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9971022009849548 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4593798816204071 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8564815521240234 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.701697051525116 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27088692784309387 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28777679800987244 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16118939220905304 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_2‚0.8186267614364624 _runtime‚6140.429679867 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25447341799736023 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17694266140460968 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2781309187412262 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998548ýgó9845275879 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.255753755569458 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2142857164144516 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3525756895542145 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0237575676292181 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16686856746673584 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.024235060438513756 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6919655799865723 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6979213953018188 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12950286269187927 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 )jet/accuracy_1_of_3‚0.930388331413269 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930713176727295 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7589252591133118 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.28097209334373474 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7611427903175354 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.3779582381248474 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.29036426544189453 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8657407164573669 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.65625Ûó9*Ê=æÄ{ particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16092664003372192 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7906768321990967 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14558538794517517 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6785169243812561 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7266062498092651 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.747317373752594 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7241339087486267 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_3‚0.6001158952713013 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961509704589844 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6643376350402832 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.263439416885376 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48003238439559937 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9921030402183533 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5562500357627869 particle/accuracy_1_of_1‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3396086096763611 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7427458763122559 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9970238208770752 % _timestamp‚1.7842789483034515e+09 )loss/val_total_loss‚8.158145491245095 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891303777694702 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6204812526702881 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992997050285339 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3396086096763611 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6551010608673096 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992997050285339 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.348013311624527 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_4‚0.9646546244621277 particle/accuracy_2_of_2‚1 _step‚593 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9165229797363281 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0634046420454979 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2673523426055908 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23579566180706024 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2553541958332062 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1525000035762787 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/sensitivity‚0.6210826635360718 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6747778654098511 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 trainer/global_step‚7199 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7687480449676514 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7918264269828796 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3387208580970764 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994879364967346 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6295425295829773 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6201350092887878 jet/accuracy_2_of_5‚1 epoch‚17 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9964581727981567 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.34725406765937805 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5859388113021851 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7147670984268188 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39732661843299866 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02193324826657772 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05475417897105217 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02193324826657772 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6687190532684326 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043982602655887604 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024154186248779297 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7975382208824158 +jet/accuracy_3_of_3‚0.06994573026895523 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02341967076063156 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6625216603279114 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9974327087402344 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5398446917533875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.34770655632019043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6110733151435852 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.714387059211731 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06757380068302155 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49099642038345337 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7821617126464844 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8484181761741638 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5577994585037231 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8484181761741638 particle/accuracy_2_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 (particle/accuracy‚0.8461061120033264 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7510585188865662 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_2_of_4‚0.7945955991744995 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7179927229881287 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9971022009849548 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4593798816204071 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8564815521240234 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.701697051525116 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27088692784309387 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.28777679800987244 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16118939220905304 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_2‚0.8186267614364624 _runtime‚6140.429679867 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25447341799736023 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17694266140460968 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2781309187412262 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985489845275879 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.255753755569458 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2142857164144516 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3525756895542145 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0237575676292181 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16686856746673584 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.024235060438513756 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6919655799865723 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6979213953018188 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12950286269187927 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 )jet/accuracy_1_of_3‚0.930388331413269 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930713176727295 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7589252591133118 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.28097209334373474 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7611427903175354 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.3779582381248474 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.29036426544189453 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8657407164573669 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.65625 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3302168846130371 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2956717908382416 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3881351053714752 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 'particle/f_score‚0.6341322064399719 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7230693101882935 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38396236300468445 +jet/accuracy_3_of_4‚0.07208774983882904 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9482510089874268 7validation_average_jet_accuracy‚0.41236504912376404 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998534619808197 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6687190532684326 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7492703199386597 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.750237762928009 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5595839023590088 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3379344344139099 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7620846033096313 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7251322865486145 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12103673070669174 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.32776179909706116 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6960147619247437 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24024946987628937 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6560530662536621 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5244975090026855 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.790650486946106 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8698999285697937 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.002801120514050126 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833432674408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962183833122253 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969032764434814 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7034876346588135 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4803662896156311 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7411195039749146 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6663458943367004 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7754723429679871 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8657407164573669 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7627173066139221 +particle/specificity‚0.9076398015022278 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39954742789268494 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925Ñ Ô |‘ç‚(:† ­ßçÒìôŒ¥ gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 <çüè‚(:ñ ­ßçÒüâô§ network.sent‚ 1.1482939e+07 network.recv‚ 1.62295e+06 disk.sda.out‚ 220.8671875 disk.dm-1.out‚ 220.8671875 disk.dm-0.in‚ 302.609375( proc.memory.availableMB‚ 741.63671875 disk.sda2.out‚0 disk.sda3.out‚ 220.8671875 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 302.609375) disk./.usagePercent‚36.61865643027026% disk.nvme0n1.out‚28781.8955078125 disk.sda1.out‚0$ memory_percent‚95.47340869903564 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 cpu‚26.013393153832414 disk.sda.in‚0 disk.rbd1.out‚98.54443359375" disk.dm-0.out‚28781.8955078125 disk.nvme0n1.in‚ 302.609375 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚28781.8955078125# proc.memory.rssMB‚ 2817.87109375$ disk./.usageGB‚545.6444435119629 disk.rbd3.in‚0* proc.memory.percent‚17.198920249938965`\¹ažé‚(:“ ¼ßçÒºý‚¥ gpu.0.gpu‚91.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù`ñê‚(:æ ¼ßçÒÍ×õ¦ disk.dm-1.in‚0 disk.dm-0.out‚ 29038.1328125( proc.memory.availableMB‚ 741.68359375 network.recv‚ 1.627466e+06 disk.sda.out‚ 221.96484375 disk.sda3.out‚ 221.96484375$ disk./.usageGB‚545.6405372619629 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 221.96484375 cpu‚26.012725468088135 disk.nvme0n1.in‚303.4375 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚303.4375# proc.memory.rssMB‚ 2817.87109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd1.out‚98.54443359375$ memory_percent‚95.47312259674072 disk.nvme0c0n1.in‚303.4375$ disk.nvme0c0n1.out‚ 29038.1328125" disk.nvme0n1.out‚ 29038.1328125* proc.memory.percent‚17.198920249938965) disk./.usagePercent‚36.61839427855521 network.sent‚ 1.1529602e+07 disk.sda.in‚0 disk.sda2.out‚0¼1¤*žë‚(:“ ËßçÒ¸’Ü£ gpu.0.gpu‚98.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00dŽýì‚(:ò ËßçÒ—°×¥ disk.sda.out‚ 223.32421875 disk.sda3.out‚ 223.32421875 disk.dm-1.in‚0' proc.memory.availableMB‚ 741.6484375# proc.memory.rssMB‚ 2817.87109375 disk.sda2.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚29045.62353515625 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚303.4375( disk.nvme0c0n1.out‚29045.62353515625 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚303.4375* proc.memory.percent‚17.198920249938965* disk./.usagePercent‚36.618392230494926 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 network.sent‚ 1.1532566e+07 disk.sda1.in‚0$ memory_percent‚95.47333717346191 cpu‚26.012022961924888# disk.dm-0.out‚29045.62353515625 disk.sda1.out‚0$ disk./.usageGB‚545.6405067443848 disk.rbd1.out‚98.60693359375 disk.dm-1.out‚ 223.32421875 disk.rbd3.in‚0 disk.nvme0n1.in‚303.4375 network.recv‚ 1.630893e+06 disk.drbd2165.out‚0†XÍ£2í- _wandbruntime‚6181  _runtime‚6181oè|´ÇîÁ & lr-AdamW/pg2‚0.0003439986526741399 trainer/global_step‚7249 % _timestamp‚1.7842789897425666e+09 _runtime‚6181.868170614 _step‚594 & lr-AdamW/pg1‚0.0003439986526741399..LÌïÆ % _timestamp‚1.7842789897425666e+09 _runtime‚6181.868170614 _step‚594 & lr-AdamW/pg1‚0.0003439986526741399 & lr-AdamW/pg2‚0.0003439986526741399 trainer/global_step‚7249Òyp>E2ð- _wandbruntime‚6181  _runtime‚6181g$²á:'ñíO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67158191 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1109936124958720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1104252511998972 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7659276846334 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57507848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67168168 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1109936124958720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1104252869588222 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1109936483337216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124676018 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57507850 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚595 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6181.869536022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12702488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250524496010240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28319717 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1096593592741888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784278989743737e+09 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124666039 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57507848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7658936296444 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1102265505521664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1109936483337216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12702491 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1096593575702528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7755513528320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67158191 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67158191 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1109936124958720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1104252511998972 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7659276846334 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57507848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67168168 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1109936124958720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1104252869588222 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1109936483337216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124676018 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57507850 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚595 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6181.869536022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12702488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250524496010240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28319717 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1096593592741888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784278989743737e+09 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124666039 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57507848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7658936296444 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1102265505521664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1109936483337216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12702491 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1096593575702528Ó}92ó- _wandbruntime‚6182  _runtime‚6182EgÞý&ô÷M J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57515480 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12704233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15619590 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57515482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250558951977472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842789903567355e+09 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28323558 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124692716 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124682723 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12704236 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7659947350166 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1110084378615296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7671649103360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67167243 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1102412746551296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚7249 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚242802306148352 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1110084737000960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7671990449664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7671649103360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1102412746551296 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15619325 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67177234 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚596 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1104400033724566 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67177234 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28323826 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7756547035648 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1096740103413760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7671990449664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57515482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250558853184000 _runtime‚6182.482526248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚242802324760576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888’3˨'õüM QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28323558 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124692716 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124682723 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12704236 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7659947350166 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1110084378615296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7671649103360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67167243 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1102412746551296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚7249 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚242802306148352 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1110084737000960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7671990449664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7671649103360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1102412746551296 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15619325 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67177234 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚596 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1104400033724566 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67177234 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28323826 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7756547035648 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1096740103413760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7671990449664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57515482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250558853184000 _runtime‚6182.482526248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚242802324760576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57515480 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12704233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15619590 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57515482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250558951977472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842789903567355e+09ÔႬæ2ö- _wandbruntime‚6182  _runtime‚6182®H“ ÷… /loss/FRt1/detection_loss‚0.4919386421449206 % _timestamp‚1.7842789903592367e+09 _step‚597 0loss/FRt2/assignment_loss‚0.9723368299790852 2loss/SRqqt2/assignment_loss‚0.5079863560983577 1loss/SRqqt1/detection_loss‚0.8694902979429268 /loss/FBt2/detection_loss‚0.8583418733819768 _runtime‚6182.484568844 0loss/FRt1/assignment_loss‚1.6914126420306432 3loss/SRqqt1/assignment_loss‚0.48855705057326304 2loss/FBt2/assignment_loss‚0.046432056598813755 .loss/FRt2/detection_loss‚1.190455614973346 1loss/SRqqt2/detection_loss‚0.6336170143788366 /loss/FBt1/detection_loss‚0.3738104575506194 3loss/FBt1/assignment_loss‚0.0022989867117787456 %loss/total_loss‚8.126677822364568 epoch‚18 trainer/global_step‚7249“ðïÌøŠ 0loss/FRt2/assignment_loss‚0.9723368299790852 2loss/SRqqt2/assignment_loss‚0.5079863560983577 1loss/SRqqt1/detection_loss‚0.8694902979429268 /loss/FBt2/detection_loss‚0.8583418733819768 _runtime‚6182.484568844 0loss/FRt1/assignment_loss‚1.6914126420306432 3loss/SRqqt1/assignment_loss‚0.48855705057326304 2loss/FBt2/assignment_loss‚0.046432056598813755 .loss/FRt2/detection_loss‚1.190455614973346 1loss/SRqqt2/detection_loss‚0.6336170143788366 /loss/FBt1/detection_loss‚0.3738104575506194 3loss/FBt1/assignment_loss‚0.0022989867117787456 %loss/total_loss‚8.126677822364568 epoch‚18 trainer/global_step‚7249 /loss/FRt1/detection_loss‚0.4919386421449206 % _timestamp‚1.7842789903592367e+09 _step‚597Õ#Eºé“ù‚(:ˆ ÚßçÒÏè‘¥ gpu.0.gpu‚84.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚202.566' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.5132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pŒJ‚øú‚(:ý ÚßçÒîµ»§ network.recv‚ 1.634278e+06$ disk./.usageGB‚545.6409301757812 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚ 223.88671875) disk.u¹f/.usagePercent‚36.61842064733123 disk.rbd0.in‚0 disk.dm-1.out‚ 223.88671875$ memory_percent‚95.47219276428223( disk.nvme0c0n1.out‚29055.24462890625 disk.rbd1.out‚98.60693359375# proc.memory.rssMB‚ 2817.87109375 disk.nvme0n1.in‚ 304.15234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 cpu‚26.011337604620522 network.sent‚ 1.1537565e+07 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 223.88671875 disk.dm-0.in‚ 304.15234375# disk.dm-0.out‚29055.24462890625' proc.memory.availableMB‚ 741.8359375* proc.memory.percent‚17.198920249938965" disk.nvme0c0n1.in‚ 304.15234375 disk.rbd3.out‚0& disk.nvme0n1.out‚29055.24462890625 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0fßYëœû‚(:‘ éßçÒÿ×¹¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚193.371' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.674200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j!!øü‚(:í éßçÒ¸ó¸¦ network.recv‚ 1.638827e+06) disk./.usagePercent‚36.61832438849836 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 224.6484375 disk.rbd3.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 741.7890625 network.sent‚ 1.1602863e+07$ disk./.usageGB‚545.6394958496094 disk.sda2.in‚0# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 cpu‚26.01067980476244 disk.drbd2165.in‚0 disk.sda.out‚ 224.6484375 disk.dm-1.in‚0 disk.rbd1.out‚98.66943359375 disk.dm-0.out‚ 29062.9921875 disk.nvme0n1.in‚ 304.98046875$ memory_percent‚95.47247886657715 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 304.98046875" disk.nvme0n1.out‚ 29062.9921875 disk.sda.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 304.98046875 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 29062.9921875 disk.sda3.out‚ 224.6484375 disk.rbd1.in‚ 0.01171875cw•;2ý- _wandbruntime‚6221  _runtime‚62217Û²÷ÇþÁ % _timestamp‚1.7842790294521754e+09 _runtime‚6221.579430437 _step‚598 & lr-AdamW/pg1‚0.0003423091257631442 & lr-AdamW/pg2‚0.0003423091257631442 trainer/global_step‚7299¸yìÌÿÆ & lr-AdamW/pg2‚0.0003423091257631442 trainer/global_step‚7299 % _timestamp‚1.7842790294521754e+09 _runtime‚6221.579430437 _step‚598 & lr-AdamW/pg1‚0.0003423091257631442ÖÌvµ2€- _wandbruntime‚6221  _runtime‚6221€¾ìô'îO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7721331978240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57889450 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12789741 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125502091 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125512070 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57889450 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1111628662079480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842790294528773e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57889448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1109627557003264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244440333928448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7709552783352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1103919109296128 _runtime‚6221.579566273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252247155563520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1117348871942144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12789738 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1117348871942144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67612643 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28512134 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15722396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976Zž•Qù'‚óO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7806924630016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1117348871942144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67612643 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28512134 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7299 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15722396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7721331978240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57889450 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12789741 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125502091 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125512070 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57889450 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1111628662079480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842790294528773e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57889448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1109627557003264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244440333928448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7709552783352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1103919109296128 _runtime‚6221.579566273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252247155563520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1117348871942144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12789738 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125528768 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252281710446080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7722344605184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7807861286400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67631686 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57897082 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7710563837074 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28515943 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842790300644674e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1117497483984384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7807947759104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1117497483984384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚6222.190303434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57897082 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125518775 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67631686 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244473744074752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125518775 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1109774780993536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1111776541394436 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_·©ô¿bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28516209 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15724723 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7710904387076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0P¾—'…ýM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28516209 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15724723 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7710904387076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125528768 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67621695 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1109774798032896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57897080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244473762686976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57897080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1109774780993536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7722685951488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12791483 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1104065637007360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252281605361152 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125528768 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252281710446080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7722344605184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7807861286400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67631686 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57897082 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7710563837074 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28515943 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842790300644674e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1117497483984384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7807947759104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1117497483984384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚6222.190303434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57897082 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125518775 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67631686 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244473744074752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125518775 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1109774780993536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1111776541394436 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ØT„'¦2†- _wandbruntime‚6222  _runtime‚6222.—- ‡ƒ 1loss/SRqqt2/detection_loss‚0.6822471248068925 %loss/total_loss‚8.180518795705416 /loss/FRt1/detection_loss‚0.5765621858339306 .loss/FBt2/detection_loss‚0.575076576428754 /loss/FRt1/assignment_loss‚1.594205740535574 trainer/global_step‚7299 % _timestamp‚1.7842790300676181e+09 _runtime‚6222.192945772 _step‚601 3loss/SRqqt1/assignment_loss‚0.45794234557276164 2loss/FBt1/assignment_loss‚0.004547223366263144 /loss/FBt1/detection_loss‚0.5422054516998792 epoch‚18 0loss/FRt2/assignment_loss‚0.8976493987245131 2loss/SRqqt2/assignment_loss‚0.5461936917131619 2loss/FBt2/assignment_loss‚0.042036599540750584 /loss/FRt2/detection_loss‚1.2807582332208836 1loss/SRqqt1/detection_loss‚0.9810942242620513Þh¡¬ˆˆ /loss/FRt1/assignment_loss‚1.594205740535574 trainer/global_step‚7299 % _timestamp‚1.7842790300676181e+09 _runtime‚6222.192945772 _step‚601 3loss/SRqqt1/assignment_loss‚0.45794234557276164 2loss/FBt1/assignment_loss‚0.004547223366263144 /loss/FBt1/detection_loss‚0.5422054516998792 epoch‚18 0loss/FRt2/assignment_loss‚0.8976493987245131 2loss/SRqqt2/assignment_loss‚0.5461936917131619 2loss/FBt2/assignment_loss‚0.042036599540750584 /loss/FRt2/detection_loss‚1.2807582332208836 1loss/SRqqt1/detection_loss‚0.9810942242620513 1loss/SRqqt2/detection_loss‚0.6822471248068925 %loss/total_loss‚8.180518795705416 /loss/FRt1/detection_loss‚0.5765621858339306 .loss/FBt2/detection_loss‚0.575076576428754Ù`û¬#‰‚(:’ øßçÒ«²Á¤ gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iL–ùŠ‚(:î øßçҪ( disk.nvme0c0n1.out‚29075.10595703125 disk.sda.out‚224.8125 proc.cpu.threads‚17& disk.nvme0n1.out‚29075.10595703125 disk.sda.in‚0( proc.memory.availableMB‚ 742.37109375# proc.memory.rssMB‚ 2817.87109375 disk.drbd2165.in‚0$ memory_percent‚95.46892642974854 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.1605928e+07 disk.sda2.out‚0 disk.rbd1.out‚98.66943359375 disk.rbd3.out‚0 disk.dm-0.in‚ 305.6953125 disk.dm-1.in‚0 disk.dm-1.out‚224.8125 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 disk.sda3.out‚224.8125# disk.dm-0.out‚29075.10595703125 network.recv‚ 1.642199e+06$ disk./.usageGB‚545.6394233703613! disk.nvme0c0n1.in‚ 305.6953125* proc.memory.percent‚17.198920249938965 disk.sda1.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 305.6953125 cpu‚26.009977715252052( disk./.usagePercent‚36.6183195243552 disk.sda2.in‚0 disk.rbd3.in‚0»(’‹‚(:‡ ‡àçÒæठgpu.0.gpu‚82.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚222.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.493 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸôÈ¡Œ‚(:û ‡àçÒ·¹í¦ disk.nvme0n1.in‚ 305.6953125& disk.nvme0n1.out‚29080.36083984375 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 305.6953125 disk.sda3.out‚ 225.05859375$ memory_percent‚95.46856880187988# proc.memory.rssMB‚ 2817.87109375( disk.nvme0c0n1.out‚29080.36083984375 disk.rbd3.in‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 742.4296875 disk.sda2.in‚0 disk.rbd1.out‚98.73193359375 disk.rbd3.out‚0 disk.sda1.in‚0 cpu‚26.009312747175485 network.sent‚ 1.1668931e+07* disk./.usagePercent‚36.618311844129174 disk.drbd2165.out‚0 disk.sda.out‚ 225.05859375 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚ 0.01171875# disk.dm-0.out‚29080.36083984375 disk.rbd0.in‚0 disk.sda.in‚0* proc.memory.percent‚17.198920249938965 network.recv‚ 1.646805e+06 disk.dm-1.out‚ 225.05859375 proc.cpu.threads‚17$ disk./.usageGB‚545.6393089294434 disk.rbd0.out‚0 disk.dm-0.in‚ 305.6953125‚'ÿC“‚(:ˆ –àçÒ‰º‡¤ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ø„øŽ‚(:í –àçÒÃò×¥ proc.cpu.threads‚17$ disk./.usageGB‚545.6393775939941 disk.rbd3.out‚0$ memory_percent‚95.46799659729004 disk.sda.in‚0 network.sent‚ 1.167398e+07 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚ 0.04296875 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 742.5234375 disk.sda3.out‚ 225.40625 disk.dm-1.out‚ 225.40625" disk.dm-0.out‚29089.6806640625% disk.nvme0n1.out‚29089.6806640625# proc.memory.rssMB‚ 2817.87109375 disk.dm-0.in‚ 306.5234375 disk.sda.out‚ 225.40625! disk.nvme0c0n1.in‚ 306.5234375 network.recv‚ 1.650081e+06 disk.drbd2165.in‚0 cpu‚26.008546751542127) disk./.usagePercent‚36.61831645226479 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 306.5234375 disk.sda1.out‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚29089.6806640625* proc.memory.percent‚17.198920249938965 disk.sda3.in‚0 disk.rbd1.out‚98.73193359375 disk.sda1.in‚0–­È2- _wandbruntime‚6261  _runtime‚6261HQ¤¤Éà trainer/global_step‚7349 % _timestamp‚1.7842790693942592e+09 _runtime‚6261.519831574 _step‚602 ' lr-AdamW/pg1‚0.00034061279227472137 ' lr-AdamW/pg2‚0.000340612792274721372NŒ Î‘È ' lr-AdamW/pg1‚0.00034061279227472137 ' lr-AdamW/pg2‚0.00034061279227472137 trainer/global_step‚7349 % _timestamp‚1.7842790693942592e+09 _runtime‚6261.519831574 _step‚602Úi3^.2’- _wandbruntime‚6261  _runtime‚6261ù=Êbô'“îO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12876991 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7858297359360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1111244659929088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚603 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7772027480064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68077072 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1111244642889728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58271050 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28704470 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246111753242624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1124761618925568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253969966219264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246111771854848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58271050 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58271048 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1124761618925568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15827482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126338143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7858212976640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚7349 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7772368819200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28704749 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1116989591445504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7760509820150 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1119005169749238 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842790693952916e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7772027480064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12876988 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7760169270260 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1124761977304064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68077072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68077072 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1111244642889728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58271050 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_sïõNKiplit_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28704470 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246111753242624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1124761618925568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253969966219264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246111771854848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58271050 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58271048 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1124761618925568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15827482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126338143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7858212976640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚7349 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7772368819200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28704749 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1116989591445504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7760509820150 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1119005169749238 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842790693952916e+09Ûâî”2•- _wandbruntime‚6262  _runtime‚6262N®9Çý&–÷M REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1124909872582144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7859223939584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126354827 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1111391153561600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68086138 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58278680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1119152691474944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15829830 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126364820 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7349 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1117136849514496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28708301 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68076147 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68076147 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1117136849514496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7761180323982 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126364820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784279070007097e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12878736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246145200613376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126354827 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7773381453312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7859316703744 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58278682 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68086138 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1111391170600960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚6262.132876639 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58278680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7773040107008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15829568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254004517317120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1124910230967808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7761520873984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12878733 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1124910230967808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7773040107008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28708566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1119152333885582‚•\'—üM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1119152691474944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15829830 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126364820 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚7349 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1117136849514496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28708301 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68076147 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68076147 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1117136849514496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7761180323982 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126364820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784279070007097e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12878736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246145200613376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126354827 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7773381453312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7859316703744 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58278682 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68086138 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1111391170600960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚6262.132876639 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58278680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7773040107008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15829568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254004517317120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1124910230967808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7761520873984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12878733 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1124910230967808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7773040107008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28708566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1119152333885582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1124909872582144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7859223939584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126354827 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1111391153561600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68086138 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58278680Ü`Ø|2˜- _wandbruntime‚6262  _runtime‚6262¤I­» ™… 0loss/FRt1/assignment_loss‚1.6528830637668703 0loss/FRt2/assignment_loss‚0.9195358650905656 2loss/SRqqt2/assignment_loss‚0.4842541705057131 1loss/FBt1/assignment_loss‚0.02255837235070842 3loss/FBt2/assignment_loss‚0.0014958956016335222 /loss/FRt2/detection_loss‚1.2831751393517576 1loss/SRqqt1/detection_loss‚1.0090170499880786 1loss/SRqqt2/detection_loss‚0.6965567597718823 2loss/SRqqt1/assignment_loss‚0.5753146739226745 0loss/FBt1/detection_loss‚0.47637972884734914 /loss/FBt2/detection_loss‚0.8274283702730348 %loss/total_loss‚8.473982759577417 epoch‚18 % _timestamp‚1.7842790700101411e+09 _step‚605 /loss/FRt1/detection_loss‚0.5253836701071499 trainer/global_step‚7349 _runtime‚6262.135468061L„E>šŠ _runtime‚6262.135468061 0loss/FRt1/assignment_loss‚1.6528830637668703 0loss/FRt2/assignment_loss‚0.9195358650905656 2loss/SRqqt2/assignment_loss‚0.4842541705057131 1loss/FBt1/assignment_loss‚0.02255837235070842 3loss/FBt2/assignment_loss‚0.0014958956016335222 /loss/FRt2/detection_loss‚1.2831751393517576 1loss/SRqqt1/detection_loss‚1.0090170499880786 1loss/SRqqt2/detection_loss‚0.6965567597718823 2loss/SRqqt1/assignment_loss‚0.5753146739226745 0loss/FBt1/detection_loss‚0.47637972884734914 /loss/FBt2/detection_loss‚0.8274283702730348 %loss/total_loss‚8.473982759577417 epoch‚18 % _timestamp‚1.7842790700101411e+09 _step‚605 /loss/FRt1/detection_loss‚0.5253836701071499 trainer/global_step‚7349Ý’„”“›‚(:ˆ ¥àçÒóìç¤ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Áß³œ‚(:ù ¥àçÒﯔ¥) disk./.usagePercent‚36.61830544394083 disk.sda3.out‚ 225.6015625 disk.dm-1.in‚0 disk.dm-0.in‚ 307.23828125 disk.drbd2165.out‚0 network.recv‚ 1.65362e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.in‚ 0.04296875 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚29096.72900390625 disk.rbd3.in‚0& disk.nvme0n1.out‚29096.72900390625# memory_percent‚95.4697847366333# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 disk.rbd1.out‚98.89208984375# disk.dm-0.out‚29096.72900390625 disk.drbd2165.in‚0 network.sent‚ 1.1678987e+07 disk.sda.out‚ 225.6015625 disk.sda2.out‚0 disk.dm-1.out‚ 225.6015625 disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 307.23828125 disk.sda.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 742.23046875$ disk./.usageGB‚545.6392135620117 disk.nvme0n1.in‚ 307.23828125 cpu‚26.007858110969913Öfí-œ‚(:‘ ´àçÒœƒæ¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚200.388' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.077600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³46ƒþž‚(:ó ´àçÒͯì¦ disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 308.06640625* proc.memory.percent‚17.198920249938965 network.sent‚ 1.1744067e+07) disk./.usagePercent‚36.61830979606891 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 225.76171875 disk.nvme0n1.in‚ 308.06640625" disk.nvme0c0n1.in‚ 308.06640625 disk.dm-0.out‚29105.50390625# proc.memory.rssMB‚ 2817.87109375 cpu‚26.00719307349325 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 225.76171875 disk.dm-1.in‚0 network.recv‚ 1.657793e+06# disk.nvme0n1.out‚29105.50390625 disk.sda3.in‚0 disk.dm-1.out‚ 225.76171875# memory_percent‚95.4695463180542$ disk./.usageGB‚545.6392784118652 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.04296875 proc.cpu.threads‚17% disk.nvme0c0n1.out‚29105.50390625 disk.rbd1.out‚98.89208984375( proc.memory.availableMB‚ 742.26953125ñ8»›Ÿ‚(: ÃàçÒÑÓ˜¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½¢ù ‚(:î ÃàçÒËà”¦ disk.dm-0.in‚ 308.78125# disk.dm-0.out‚29113.05419921875 disk.rbd0.in‚0 network.recv‚ 1.661581e+06 disk.sda2.in‚0 disk.nvme0n1.in‚ 308.78125& disk.nvme0n1.out‚29113.05419921875 disk.sda.out‚ 226.171875# proc.memory.rssMB‚ 2817.87109375$ disk./.usageGB‚545.6371803283691 disk.sda3.out‚ 226.171875 disk.sda1.in‚0$ memory_percent‚96.97282314300537 disk.rbd0.out‚0* disk./.usagePercent‚36.618168991925074 disk.sda.in‚0 disk.rbd1.in‚ 0.04296875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚29113.05419921875 disk.dm-1.out‚ 226.171875 disk.drbd2165.out‚0 cpu‚26.0065515849169 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 495.97265625* proc.memory.percent‚17.198920249938965 network.sent‚ 1.1749119e+07 disk.rbd1.out‚99.35009765625 disk.nvme0c0n1.in‚ 308.78125 disk.sda3.in‚0 disk.rbd3.out‚0lŠû 2¡- _wandbruntime‚6301  _runtime‚6301Ɖ;oɢà _runtime‚6301.252475449 _step‚606 ' lr-AdamW/pg1‚0.00033890976116120164 ' lr-AdamW/pg2‚0.00033890976116120164 trainer/global_step‚7399 % _timestamp‚1.7842791091268647e+09“Á$Î£È ' lr-AdamW/pg1‚0.00033890976116120164 ' lr-AdamW/pg2‚0.00033890976116120164 trainer/global_step‚7399 % _timestamp‚1.7842791091268647e+09 _runtime‚6301.252475449 _step‚606ÞðP2¤- _wandbruntime‚6301  _runtime‚6301›íPÛ¥îO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7810785757168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58652650 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1126380962240496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7811126307058 _runtime‚6_UϪ"301.253723544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68531524 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127184174 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15932611 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68521547 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28896852 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1132174724287488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7399 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255692644788736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7822722981888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68531524 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1124351642927104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7909542196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28896581 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1124351659966464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58652650 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12964241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247783209781248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1132174724287488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7909453619712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58652648 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1132174365908992 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1124351659966464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7823064321024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12964238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247783191169024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127174195 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1124351642927104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1126381319829746 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1118570193522688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7822722981888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58652648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1132174365908992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255692751977984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7823064321024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15932343 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842791091279237e+09 _step‚607,)hËù'¦óO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68521547 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28896852 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1132174724287488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7399 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255692644788736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7822722981888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68531524 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1124351642927104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7909542196736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28896581 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1124351659966464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58652650 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12964241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247783209781248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1132174724287488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7909453619712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58652648 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1132174365908992 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1124351659966464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7823064321024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12964238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247783191169024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127174195 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1124351642927104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1126381319829746 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1118570193522688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7822722981888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58652648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1132174365908992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255692751977984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7823064321024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15932343 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842791091279237e+09 _step‚607 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7810785757168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58652650 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1126380962240496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7811126307058 _runtime‚6301.253723544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68531524 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127184174 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15932611ßôxi2§- _wandbruntime‚6302  _runtime‚63027ªÁ¿þ&¨øM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7910469685760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127200872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚6302.062167355 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7823735608832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28900696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58660282 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68540590 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7823735608832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7811796810890 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68540590 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127190879 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68530599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68530599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1124498900996096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255727089613312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚7399 % _timestamp‚1.7842791099363093e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28900419 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255727198892544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58660282 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12965986 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1126528483966090 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127200872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1118716704194560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58660280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127190879 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247816619927552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7812137360892 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7824076955136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7910560352768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127200872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚6302.062167355 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7823735608832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28900696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58660282 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68540590 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7823735608832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7811796810890 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68540590 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127190879 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68530599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68530599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1124498900996096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255727089613312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚7399 % _timestamp‚1.7842791099363093e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28900419 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255727198892544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58660282 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12965986 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1126528483966090 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127200872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1118716704194560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58660280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127190879 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247816619927552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7812137360892 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7824076955136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7910560352768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59034250 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1131713694408704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7960696505344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1139587112892416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚611 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1131713711448064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16037085 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚257415239120896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1131713694408704ú,cÎø'¶òO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13051488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1131713711448064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16037085 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚257415239120896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1131713694408704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7873418483712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13051491 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249454629095424 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7873418483712 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1139587112892416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257415344212992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1133757469910254 trainer/global_step‚7449 _runtime‚6341.262607737 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68975999 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7861742793966 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68985976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59034248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1133757112321004 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1125895727116288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7873759822848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128020226 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68985976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59034250 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1131713694408704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7960696505344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1139587112892416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚611 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7862413297798 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128036924 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚6341.874122237 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59041882 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1133904634046598 _step‚612 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7961719742976 % _timestamp‚1.7842791497483516e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1131860952477696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59041880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59041882 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13053233 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7874431110656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68995042 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68995042 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1139735366548992 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1126042237788160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7961635367424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29092017 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7862753847800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16038784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29092017 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7862753847800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16038784 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7862413297798 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128036924 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚6341.874122237 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59041882 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1133904634046598 _step‚612 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7961719742976 % _timestamp‚1.7842791497483516e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1131860952477696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59041880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59041882 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13053233 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7874431110656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68995042 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68995042äį¥2º- _wandbruntime‚6341  _runtime‚6341&ãWÑ »ƒ /loss/FRt2/assignment_loss‚0.884886395430913 2loss/SRqqt2/assignment_loss‚0.5470217576441823 /loss/FRt1/detection_loss‚0.5291878279401322 /loss/FBt1/detection_loss‚0.4455062816340993 trainer/global_step‚7449 % _timestamp‚1.7842791497515745e+09 _runtime‚6341.876908677 0loss/FRt1/assignment_loss‚1.6855765330986898 /loss/FBt2/detection_loss‚0.8491046134149656 2loss/SRqqt1/assignment_loss‚0.5478799182628963 2loss/FBt2/assignment_loss‚0.013633588563418675 1loss/SRqqt2/detection_loss‚0.5847840881171349 2loss/FBt1/assignment_loss‚0.006137648481436477 /loss/FRt2/detection_loss‚1.2072988440126289 1loss/SRqqt1/detection_loss‚0.9123178184897868 %loss/total_loss‚8.213335315090283 epoch‚18 _step‚613SŸ‚Ó¼ˆ _step‚613 /loss/FRt2/assignment_loss‚0.884886395430913 2loss/SRqqt2/assignment_loss‚0.5470217576441823 /loss/FRt1/detection_loss‚0.5291878279401322 /loss/FBt1/detection_loss‚0.4455062816340993 trainer/global_step‚7449 % _timestamp‚1.7842791497515745e+09 _runtime‚6341.876908677 0loss/FRt1/assignment_loss‚1.6855765330986898 /loss/FBt2/detection_loss‚0.8491046134149656 2loss/SRqqt1/assignment_loss‚0.5478799182628963 2loss/FBt2/assignment_loss‚0.013633588563418675 1loss/SRqqt2/detection_loss‚0.5847840881171349 2loss/FBt1/assignment_loss‚0.006137648481436477 /loss/FRt2/detection_loss‚1.2072988440126289 1loss/SRqqt1/detection_loss‚0.9123178184897868 %loss/total_loss‚8.213335315090283 epoch‚18åSoE‘½‚(:† ðàçÒîæ½¥ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k_×¢ÿ¾‚(:ô ðàçҋ᧠network.sent‚ 1.1833291e+07# proc.memory.rssMB‚ 2817.87109375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 310.32421875$ memory_percent‚96.97468280792236( proc.memory.availableMB‚ 495.66796875$ disk.nvme0n1.out‚29138.931640625 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚29138.931640625) disk./.usagePercent‚36.61850256974219 disk.nvme0n1.in‚ 310.32421875 disk.sda.out‚ 227.11328125 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0! disk.dm-0.out‚29138.931640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 227.11328125 cpu‚26.004535879846827$ disk./.usageGB‚545.6421508789062 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd1.out‚99.47509765625 network.recv‚ 1.672756e+06" disk.nvme0c0n1.in‚ 310.32421875 disk.sda.in‚0 disk.sda3.out‚ 227.11328125* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17Lû’¿‚(:‡ ÿàçÒƯԣ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬bŠõÀ‚(:ê ÿàçÒ”ÊÒ¥ disk.sda2.out‚0 disk.dm-1.out‚ 227.34375( proc.memory.availableMB‚ 496.11328125 cpu‚26.003879695702313$ disk.nvme0n1.out‚29147.517578125 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17$ disk./.usageGB‚545.6421203613281 disk.sda3.out‚ 227.34375 disk.sda1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚29147.517578125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2817.87109375) disk./.usagePercent‚36.61850052168191 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 227.34375 disk.dm-0.in‚ 311.15234375! disk.dm-0.out‚29147.517578125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 311.15234375 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 network.recv‚ 1.676773e+06 disk.nvme0n1.in‚ 311.15234375 disk.rbd0.out‚0# memory_percent‚96.9719648361206 network.sent‚ 1.1884453e+07 disk.rbd1.out‚99.53759765625 disk.rbd3.in‚04n’Á‚(:‡ ŽáçÒþ¢ˆ¥ gpu.0.gpu‚52.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚213.355' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.671 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;AJö‚(:ë ŽáçҘﰧ disk.sda1.out‚0 disk.rbd0.in‚0 network.sent‚ 1.1889445e+07 disk.sda3.out‚ 227.6953125 disk.rbd3.in‚0( proc.memory.availableMB‚ 495.87109375# disk./.usageGB‚545.641242980957 disk.sda3.in‚0 disk.rbd1.out‚99.53759765625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 311.15234375 disk.rbd0.out‚0% disk.nvme0c0n1.out‚29154.50390625$ memory_percent‚96.97344303131104 network.recv‚ 1.67994e+06 disk.sda2.out‚0 disk.dm-0.out‚29154.50390625 disk.nvme0n1.in‚ 311.15234375 disk.drbd2165.out‚0 disk.dm-1.out‚ 227.6953125# disk.nvme0n1.out‚29154.50390625 disk.sda.out‚ 227.6953125) disk./.usagePercent‚36.61844163994904 disk.rbd1.in‚0.0625" disk.nvme0c0n1.in‚ 311.15234375# proc.memory.rssMB‚ 2817.87109375* proc.memory.percent‚17.198920249938965 cpu‚26.00323295388951 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0ù”sm2Ã- _wandbruntime‚6381  _runtime‚6381e,/1ÈÄ ' lr-AdamW/pg2‚0.00033548404401204606 trainer/global_step‚7499 % _timestamp‚1.7842791890487125e+09 _runtime‚ 6381.17443057 _step‚614 ' lr-AdamW/pg1‚0.00033548404401204606‚ú4ÍÅÇ ' lr-AdamW/pg1‚0.00033548404401204606 ' lr-AdamW/pg2‚0.00033548404401204606 trainer/global_step‚7499 % _timestamp‚1.7842791890487125e+09 _runtime‚ 6381.17443057 _step‚614æe ز2Æ- _wandbruntime‚6381  _runtime‚6381õ°éLKÇíO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1147000218254336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1141133619990762 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69440428 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7912018730984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128846299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1139075762929664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69430451 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784279189049855e+09 _step‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13138741 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1133221260709888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1146999859875840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7924455324672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1139075762929664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59415848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8011870479872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_starΰͨ t/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1133221243670528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128856278 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69440428 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29280245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259137956113920 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1139075745890304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7912359280874 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13138738 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1146999859875840 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1139075745890304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59415850 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16141238†EþÝø'ÈòO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13138738 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1146999859875840 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1139075745890304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59415850 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16141238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1147000218254336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1141133619990762 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69440428 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7912018730984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128846299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1139075762929664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69430451 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784279189049855e+09 _step‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13138741 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1133221260709888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1146999859875840 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7924455324672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1139075762929664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59415848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8011870479872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1133221243670528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128856278 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69440428 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29280245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259137956113920 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1139075745890304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7912359280874 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088瀡U 2É- _wandbruntime‚6381  _runtime‚6381Ûæ3ý&Ê÷M E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7925126612480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128862983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1133367754342400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69439503 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784279189661534e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1139222986919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7913370334708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1147148113532416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251159514392576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59423480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1139223003959296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16143559 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251159495780352 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29283791 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259172289792000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1147148471918080 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7925467958784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29284045 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259172401168384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13140486 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128872976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128872976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128862983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13140483 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59423480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8012886775808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16143308 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1133367771381760 _step‚616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69449494 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1139223003959296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7925467958784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59423482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69439503 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ×0{%ËüM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59423482 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1139222986919936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7913029784706 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1147148471918080 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7925467958784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29284045 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259172401168384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13140486 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128872976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128872976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128862983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13140483 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59423480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8012886775808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16143308 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1133367771381760 _step‚616 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69449494 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1139223003959296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7925467958784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59423482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69439503 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7925126612480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128862983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1133367754342400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69439503 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784279189661534e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1139222986919936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7913370334708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1147148113532416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251159514392576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59423480 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1139223003959296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16143559 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251159495780352 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29283791 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259172289792000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 QDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1154412606859264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7974809487360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69884903 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7962975767782 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1154412606859264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252797504948224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260860478039040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7975150826496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129692330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29471873 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252797523560448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7975150826496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1140546777264128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088-È´jù'ÚóO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252797504948224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260860478039040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7975150826496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129692330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29471873 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252797523560448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7975150826496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1140546777264128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1146437814411264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1146437797371904 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1154412606859264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7974809487360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69884903 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7962975767782 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1154412606859264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ëÞíd2Û- _wandbruntime‚6421  _runtime‚6421ÂòœuÜøM QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1146585055440896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7963646271614 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129699035 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252830952318976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1146585055440896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7975822114304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7975822114304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59805080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252830933706752 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7842792293327742e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69893955 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59805080 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1140693304975360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129709028 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1140693287936000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7976163460608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13227733 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16247676 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1148657291796976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69893955 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29475409 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260895035543040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129709028 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1140693287936000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7976163460608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13227733 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16247676 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1148657291796976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69893955 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29475409 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260895035543040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1146585055440896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7963646271614 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129699035 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252830952318976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1146585055440896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7975822114304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7975822114304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59805080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252830933706752 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7842792293327742e+09ìBtÏÐ2Þ- _wandbruntime‚6421  _runtime‚6421åO« ߃ 0loss/FRt2/assignment_loss‚0.9176157832605959 2loss/FBt1/assignment_loss‚0.007733199519056077 1loss/SRqqt2/detection_loss‚0.6755569419698336 0loss/FBt1/detection_loss‚0.46453544789658885 /loss/FBt2/detection_loss‚0.7878967161964754 2loss/SRqqt2/assignment_loss‚0.4983937147938319 2loss/FBt2/assignment_loss‚0.000719756029799759 /loss/FRt1/detection_loss‚0.5254417677015739 /loss/FRt2/detection_loss‚1.3004145813821797 trainer/global_step‚7549 $ _timestamp‚1.784279229335896e+09 _runtime‚6421.461258847 0loss/FRt1/assignment_loss‚1.6136435251901198 2loss/SRqqt1/assignment_loss‚0.5214968067059655 0loss/SRqqt1/detection_loss‚0.886315017088006 _step‚621 %loss/total_loss‚8.199763257734027 epoch‚18Öç;àˆ 0loss/FRt2/assignment_loss‚0.9176157832605959 2loss/FBt1/assignment_loss‚0.007733199519056077 1loss/SRqqt2/detection_loss‚0.6755569419698336 0loss/FBt1/detection_loss‚0.46453544789658885 /loss/FBt2/detection_loss‚0.7878967161964754 2loss/SRqqt2/assignment_loss‚0.4983937147938319 2loss/FBt2/assignment_loss‚0.000719756029799759 /loss/FRt1/detection_loss‚0.5254417677015739 /loss/FRt2/detection_loss‚1.3004145813821797 trainer/global_step‚7549 $ _timestamp‚1.784279229335896e+09 _runtime‚6421.461258847 0loss/FRt1/assignment_loss‚1.6136435251901198 2loss/SRqqt1/assignment_loss‚0.5214968067059655 0loss/SRqqt1/detection_loss‚0.886315017088006 _step‚621 %loss/total_loss‚8.199763257734027 epoch‚18íPr‘á‚(:† ÊáçÒ‚šŽ¥ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ,µ;â‚(:ù ÊáçÒÙµ²§* proc.memory.percent‚17.198920249938965) disk./.usagePercent‚36.61817052797028" disk.nvme0c0n1.in‚ 314.23828125 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2817.87109375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 229.01171875 cpu‚26.000541195369152 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 229.01171875 disk.sda2.in‚0 disk.sda3.out‚ 229.01171875# disk.dm-0.out‚29187.95458984375$ memory_percent‚96.97210788726807 disk.nvme0n1.in‚ 314.23828125 disk.rbd3.in‚0$ disk./.usageGB‚545.6372032165527( disk.nvme0c0n1.out‚29187.95458984375 disk.dm-0.in‚ 314.23828125 network.recv‚ 1.69428e+06& disk.nvme0n1.out‚29187.95458984375 disk.rbd1.out‚99.70947265625 proc.cpu.threads‚17 network.sent‚ 1.1965507e+07 disk.dm-1.in‚0( proc.memory.availableMB‚ 496.08984375 • ’“ã‚(:ˆ ÙáçÒޥä gpu.0.gpu‚68.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚237.593' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5186 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N)üä‚(:ñ ÙáçÒàÑç¦ disk.nvme0n1.in‚ 314.23828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 496.14453125* disk./.usagePercent‚36.618554795279174& disk.nvme0n1.out‚29191.42822265625 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚29191.42822265625 disk.sda2.out‚0 disk.sda3.out‚ 229.28125 disk.dm-1.out‚ 229.28125 disk.rbd1.out‚99.77197265625 disk.dm-0.in‚ 314.23828125# proc.memory.rssMB‚ 2817.87109375 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd3.in‚0$ memory_percent‚96.97177410125732* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17 network.sent‚ 1.2030786e+07" disk.nvme0c0n1.in‚ 314.23828125 disk.sda.out‚ 229.28125 disk.drbd2165.in‚0 cpu‚25.999928977238845$ disk./.usageGB‚545.6429290771484 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚29191.42822265625 network.recv‚ 1.69924e+06Æõ-2å- _wandbruntime‚6460  _runtime‚6460 ÔñŒÇæÁ & lr-AdamW/pg1‚0.0003320328544111895 & lr-AdamW/pg2‚0.0003320328544111895 trainer/global_step‚7599 % _timestamp‚1.7842792686310241e+09 _runtime‚6460.757370389 _step‚6220Ÿ×ÌçÆ & lr-AdamW/pg1‚0.0003320328544111895 & lr-AdamW/pg2‚0.0003320328544111895 trainer/global_step‚7599 % _timestamp‚1.7842792686310241e+09 _runtime‚6460.757370389 _step‚622î2¹S2è- _wandbruntime‚6460  _runtime‚6460:“mô'éîO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262583382527488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1153799865892864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70349332 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1155885562562528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60179050 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1147872310857728 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1153799865892864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8025846328320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1161825353842688 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1161825353842688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13313238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8114321977856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1155885920151778 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8025846328320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13313241 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16350255 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70339355 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70349332 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1161825712221184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8025504989184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60179048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚254468942874624 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1153799848853504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70339355 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1161825712221184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8025504989184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262583264852480 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1147872327897088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1153799848853504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚6460.757646511 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60179050 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8013592254690 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842792686317918e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130528382 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8114421040640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8013251704800 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚623 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29663496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29663215 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60179048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚254468961486848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚7599 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0(m[³ êóO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1161825353842688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13313238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8114321977856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1155885920151778 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8025846328320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13313241 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16350255 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70339355 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70349332 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1161825712221184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8025504989184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60179048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚254468942874624 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1153799848853504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70339355 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1161825712221184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8025504989184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262583264852480 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1147872327897088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1153799848853504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split@.ýF_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚6460.757646511 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60179050 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8013592254690 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842792686317918e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130528382 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8114421040640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8013251704800 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚623 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29663496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29663215 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60179048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚254468961486848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚7599 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262583382527488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1153799865892864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70349332 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1155885562562528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60179050 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1147872310857728 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1153799865892864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8025846328320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1161825353842688 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8014262758522 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚7599 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1161973965884928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29667000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70348407 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8026858962432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13314986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8115345726464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1148018838568960 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1148018821529600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8026858962432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70358398 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8014603308524 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262617717359616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8115434296320 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842792692454524e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1153947106922496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚254502371633152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60186682 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1156033441877484 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60186682 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70358398 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60186680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16352287 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130535087 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262617824541696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130545080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13314983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16352017 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130545080 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1161973965884928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8026517616128 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1153947089883136 _step‚624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29667273 Q'íýM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60186682 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1156033441877484 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60186682 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70358398 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60186680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16352287 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130535087 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262617824541696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130545080 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13314983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16352017 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130545080 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1161973965884928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8026517616128 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1153947089883136 _step‚624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29667273 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8014262758522 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚7599 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1161973965884928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29667000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70348407 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8026858962432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13314986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8115345726464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1148018838568960 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1148018821529600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8026858962432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70358398 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8014603308524 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262617717359616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8115434296320 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842792692454524e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1153947106922496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚254502371633152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616ð;×ëÔ2î- _wandbruntime‚6461  _runtime‚6461¦3 dï‚ /loss/FBt2/detection_loss‚0.8196112291059561 trainer/global_step‚7599 % _timestamp‚1.7842792692472584e+09 /loss/FRt1/assignment_loss‚1.614392763748357 3loss/FBt2/assignment_loss‚0.0018718658911672668 /loss/FRt2/detection_loss‚1.2433446007465396 2loss/SRqqt2/assignment_loss‚0.5500563059743375 epoch‚18 2loss/FBt1/assignment_loss‚0.030924498812707868 /loss/FRt1/detection_loss‚0.5548748748296711 1loss/SRqqt2/detection_loss‚0.7197027324443035 %loss/total_loss‚8.441319885467475 _runtime‚6461.372584035 _step‚625 /loss/FRt2/assignment_loss‚0.929084108735486 1loss/SRqqt1/assignment_loss‚0.505119462020784 1loss/SRqqt1/detection_loss‚0.9439195762065872 /loss/FBt1/detection_loss‚0.5284178669515774;[ ð‡ /loss/FRt2/assignment_loss‚0.929084108735486 1loss/SRqqt1/assignment_loss‚0.505119462020784 1loss/SRqqt1/detection_loss‚0.9439195762065872 /loss/FBt1/detection_loss‚0.5284178669515774 /loss/FBt2/detection_loss‚0.8196112291059561 trainer/global_step‚7599 % _timestamp‚1.7842792692472584e+09 /loss/FRt1/assignment_loss‚1.614392763748357 3loss/FBt2/assignment_loss‚0.0018718658911672668 /loss/FRt2/detection_loss‚1.2433446007465396 2loss/SRqqt2/assignment_loss‚0.5500563059743375 epoch‚18 2loss/FBt1/assignment_loss‚0.030924498812707868 /loss/FRt1/detection_loss‚0.5548748748296711 1loss/SRqqt2/detection_loss‚0.7197027324443035 %loss/total_loss‚8.441319885467475 _runtime‚6461.372584035 _step‚625ñv—sñ‚(:’ èáçÒñ›¾¥ gpu.0.gpu‚10.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÍ6Wüò‚(:ñ èáçÒ–®è§# disk./.usageGB‚545.642936706543& disk.nvme0n1.out‚29200.06494140625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.198920249938965 disk.nvme0c0n1.in‚ 314.953125 disk.sda2.in‚0 network.sent‚ 1.2046791e+07 disk.rbd1.in‚0.0625 disk.rbd0.in‚0 disk.rbd1.out‚99.77197265625 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 229.42578125* disk./.usagePercent‚36.618555307294244 disk.dm-1.in‚0# proc.memory.rssMB‚ 2817.87109375 network.recv‚ 1.702362e+06& proc.memory.availableMB‚ 496.015625( disk.nvme0c0n1.out‚29200.06494140625 disk.rbd3.in‚0 disk.dm-0.in‚ 314.953125# disk.dm-0.out‚29200.06494140625 disk.nvme0n1.in‚ 314.953125 disk.sda1.out‚0$ memory_percent‚96.97256088256836 cpu‚26.00126468790082 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 229.42578125 disk.sda3.out‚ 229.42578125 proc.cpu.threads‚17K¹52ó- _wandbruntime‚6478  _runtime‚6478ó  ƒ(ô±{ @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8101851940155029 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8690986633300781 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6664990186691284 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5273316502571106 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7243229746818542 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7867671251296997 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3182547688484192 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1691666692495346 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_3_of_4‚0.08556707948446274 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7845219969749451 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7166091799736023 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9954127073287964 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7086498141288757 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/*FRt1SRqqt0FBt/FRÂàÇ5t_purity‚0.17321425676345825 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_5‚1 trainer/global_step‚7599 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7872637510299683 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6521817445755005 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7518832683563232 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 (loss/val_total_loss‚8.10343289490091 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9810983538627625 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28822091221809387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_5_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7965230345726013 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26536881923675537 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36143264174461365 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9061917662620544 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987881183624268 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18905825912952423 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6928094029426575 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7420092821121216 *jet/accuracy_1_of_2‚0.8083617091178894 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,jet/accuracy_4_of_4‚0.011951448395848274 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2502525746822357 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7144590020179749 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3031337261199951 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6752033233642578 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5124417543411255 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3193291127681732 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.799357533454895 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04703608155250549 )jet/accuracy_2_of_2‚0.165538489818573 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5808226466178894 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7925089001655579 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9248695373535156 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.506336510181427 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6929945349693298 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7057989239692688 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7839100360870361 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6638073921203613 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963633418083191 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2339351922273636 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850064516067505 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7288532257080078 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_3‚0.5873154401779175 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 (particle/accuracy‚0.8459975719451904 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3076874315738678 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23461316525936127 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24307377636432648 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5829229950904846 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7993953227996826 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7287319898605347 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.026734450832009315 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7059807777404785 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6594350934028625 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03903000429272652 _step‚626 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7548428773880005 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1604701280593872 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4619158208370209 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120370149612427 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5929864048957825 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.8062357306480408 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9846278429031372 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09049393981695175 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6664990186691284 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7147229909896851 +jet/accuracy_1_of_1‚0.36561208963394165 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 particle/accuracy_2_of_3‚1 7validation_average_jet_accuracy‚0.40216517448425293 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27090293169021606 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30951759219169617 &particle/f_score‚0.630793571472168 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +particle/sensitivity‚0.6127094030380249 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.273954302072525 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.368825763463974 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02795976959168911 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_5_of_5‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30951759219169617 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05468783900141716 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6468985676765442 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02568543516099453 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3355478048324585 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23276446759700775 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842792863395112e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39828068017959595 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26311731338500977 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23031584918498993 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6932687163352966 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6518896818161011 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5007466077804565 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8594176769256592 *jet/accuracy_1_of_3‚0.9279488325119019 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.726568341255188 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25971293449401855 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31968262791633606 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _runtime‚6478.465763141 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24708418548107147 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3957863450050354 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30889734625816345 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02568543516099453 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02751920558512211 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6647911071777344 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +particle/specificity‚0.9097923040390015 epoch‚18 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2994025945663452 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7638416290283203 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6299375295639038 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8594176769256592 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17613494396209717 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07326029986143112 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8644052147865295 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6603437662124634 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04329235106706619 /particle/accuracy_4_of_4‚0.9370694756507874 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7436873316764832 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5052391290664673 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6615539789199829 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027457334101200104 /particle/accuracy_3_of_4‚0.9946429133415222 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7570406794548035 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.672008216381073 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5399999618530273 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6918128132820129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9912953972816467 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7925548553466797 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8101851940155029 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9950835108757019 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07513906061649323 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916428923606873 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129aRÀ]¼=õ¶{ )jet/accuracy_2_of_2‚0.165538489818573 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5808226466178894 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7925089001655579 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9248695373535156 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.506336510181427 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6929945349693298 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7057989239692688 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7839100360870361 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6638073921203613 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963633418083191 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2339351922273636 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850064516067505 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7288532257080078 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_3‚0.5873154401779175 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 (particle/accuracy‚0.8459975719451904 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3076874315738678 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23461316525936127 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24307377636432648 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5829229950904846 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7993953227996826 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7287319898605347 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.026734450832009315 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7059807777404785 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6594350934028625 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03903000429272652 _step‚626 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7548428773880005 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1604701280593872 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4619158208370209 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120370149612427 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5929864048957825 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.8062357306480408 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9846278429031372 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09049393981695175 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6664990186691284 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7147229909896851 +jet/accuracy_1_of_1‚0.36561208963394165 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 particle/accuracy_2_of_3‚1 7validation_average_jet_accuracy‚0.40216517448425293 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27090293169021606 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30951759219169617 &particle/f_score‚0.630793571472168 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +particle/sensitivity‚0.6127094030380249 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.273954302072525 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.368825763463974 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02795976959168911 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_5_of_5‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30951759219169617 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05468783900141716 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6468985676765442 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02568543516099453 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3355478048324585 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23276446759700775 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842792863395112e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39828068017959595 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26311731338500977 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23031584918498993 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6932687163352966 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6518896818161011 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5007466077804565 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8594176769256592 *jet/accuracy_1_of_3‚0.9279488325119019 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.726568341255188 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25971293449401855 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31968262791633606 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 _runtime‚6478.465763141 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24708418548107147 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3957863450050354 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30889734625816345 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02568543516099453 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02751920558512211 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6647911071777344 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +particle/specificity‚0.9097923040390015 epoch‚18 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2994025945663452 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7638416290283203 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6299375295639038 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8594176769256592 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17613494396209717 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07326029986143112 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8644052147865295 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6603437662124634 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04329235106706619 /particle/accuracy_4_of_4‚0.9370694756507874 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7436873316764832 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5052391290664673 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6615539789199829 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027457334101200104 /particle/accuracy_3_of_4‚0.9946429133415222 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7570406794548035 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.672008216381073 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5399999618530273 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6918128132820129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9912953972816467 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7925548553466797 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8101851940155029 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9950835108757019 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07513906061649323 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916428923606873 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8101851940155029 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8690986633300781 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6664990186691284 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5273316502571106 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7243229746818542 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7867671251296997 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3182547688484192 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1691666692495346 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_3_of_4‚0.08556707948446274 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7845219969749451 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7166091799736023 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9954127073287964 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7086498141288757 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17321425676345825 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_5‚1 trainer/global_step‚7599 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7872637510299683 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6521817445755005 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7518832683563232 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 (loss/val_total_loss‚8.10343289490091 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9810983538627625 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28822091221809387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_5_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7965230345726013 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26536881923675537 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36143264174461365 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9061917662620544 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987881183624268 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18905825912952423 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6928094029426575 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7420092821121216 *jet/accuracy_1_of_2‚0.8083617091178894 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,jet/accuracy_4_of_4‚0.011951448395848274 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2502525746822357 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7144590020179749 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3031337261199951 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6752033233642578 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5124417543411255 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3193291127681732 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.799357533454895 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04703608155250549ò‰è¸Hö‚(:’ ÷áçÒæÊû¤ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚262.359' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.47179999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,XFÛü÷‚(:ñ ÷áçÒžëç& disk.nvme0n1.out‚29206.99658203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 315.78125 disk.sda.in‚0 cpu‚26.008964850400336 network.recv‚ 1.706931e+06* disk./.usagePercent‚36.618546091023006 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 229.69921875 disk.nvme0n1.in‚ 315.78125( disk.nvme0c0n1.out‚29206.99658203125 disk.sda.out‚ 229.69921875 disk.sda3.in‚0$ memory_percent‚96.97365760803223* proc.memory.percent‚17.198920249938965$ disk./.usageGB‚545.6427993774414# disk.dm-0.out‚29206.99658203125 disk.sda2.out‚0 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2817.87109375 disk.rbd3.in‚0 disk.dm-1.out‚ 229.69921875 disk.rbd1.out‚99.97509765625 network.sent‚ 1.2096745e+07 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 315.78125 disk.drbd2165.in‚0' proc.memory.availableMB‚ 495.8359375 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17ù.¸Êœø‚(:‘ †âçÒäÃ¥ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô1@þù‚(:ó †âçÒÊ­§" disk.nvme0c0n1.in‚ 316.49609375 disk.dm-1.out‚ 229.8359375 disk.dm-0.in‚ 316.49609375 disk.sda1.out‚0 disk.drbd2165.in‚0 network.sent‚ 1.2156266e+07 disk.sda.out‚ 229.8359375 disk.rbd3.in‚0* proc.memory.percent‚17.198920249938965 cpu‚26.008333988031062 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚29214.8037109375 disk.sda3.out‚ 229.8359375% disk.nvme0n1.out‚29214.8037109375# memory_percent‚96.9733476638794( proc.memory.availableMB‚ 495.88671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 proc.cpu.threads‚17 network.recv‚ 1.711274e+06$ disk./.usageGB‚545.6428489685059 disk.rbd1.out‚99.97509765625 disk.rbd3.out‚0" disk.dm-0.out‚29214.8037109375 disk.nvme0n1.in‚ 316.49609375# proc.memory.rssMB‚ 2817.87109375) disk./.usagePercent‚36.61854941912095‡¡I³Qú‚(:„ •âçÒÞØÚ¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.> (µ>correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÚäþû‚(:ó •âçÒ ¯ü¦ network.sent‚ 1.2161374e+07( disk.nvme0c0n1.out‚29222.38818359375 disk.sda.out‚ 230.140625 disk.dm-0.in‚ 317.32421875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚0) disk./.usagePercent‚36.61853789878191 disk.rbd0.out‚0 disk.dm-1.in‚0 cpu‚26.00766974212031 network.recv‚ 1.714959e+06 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 317.32421875 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.dm-1.out‚ 230.140625' proc.memory.availableMB‚ 495.8359375# proc.memory.rssMB‚ 2817.87109375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.198920249938965$ disk./.usageGB‚545.6426773071289 disk.sda3.out‚ 230.140625! disk.rbd1.out‚100.22705078125 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda.in‚0# disk.dm-0.out‚29222.38818359375 disk.nvme0n1.in‚ 317.32421875& disk.nvme0n1.out‚29222.38818359375$ memory_percent‚96.97365760803223©u™Ô2ü- _wandbruntime‚6517  _runtime‚6517‚çõÆýÀ & lr-AdamW/pg2‚0.0003302979842668369 trainer/global_step‚7649 $ _timestamp‚1.784279325652363e+09 _runtime‚6517.777949005 _step‚627 & lr-AdamW/pg1‚0.0003302979842668369Èq6lËþÅ & lr-AdamW/pg1‚0.0003302979842668369 & lr-AdamW/pg2‚0.0003302979842668369 trainer/global_step‚7649 $ _timestamp‚1.784279325652363e+09 _runtime‚6517.777949005 _step‚627óÝž2ÿ- _wandbruntime‚6517  _runtime‚65174‘]2ô'€îO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8094275035648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1171185759060480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70874106 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29888735 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29888468 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842793256533742e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8081928466827 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1171185400681984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8183974460416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256156292900864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70874106 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1163091484024832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13403008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1171185400681984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13403011 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131555422 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚6517.782368208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8093933696512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8183883786240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70864129 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264340176687104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0*.›Áù'óO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29888735 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29888468 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842793256533742e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8081928466827 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1171185400681984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8183974460416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256156292900864 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70874106 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1163091484024832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13403008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1171185400681984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13403011 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131555422 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚6517.782368208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8093933696512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8183883786240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70864129 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264340176687104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8094275035648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1171185759060480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70874106ôz·c72‚- _wandbruntime‚6518  _runtime‚6518òæyÓþ&ƒøM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1157253035376640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8082939520661 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1171333654338560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264374735911424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8082598970659 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131572120 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1163238708015104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8095287669760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29892322 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13404753 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131572120 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131562127 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264374630826496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚629 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842793262684433e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70883172 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29892588 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1165335617307939 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256189721659392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16487569 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60688946 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1163238708015104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8095287669760 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60688946 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6518.394383933 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184995639808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8094946323456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131562127 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70873181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60688948 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8094946323456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16487569 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60688946 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1163238708015104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8095287669760 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60688946 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6518.394383933 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184995639808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8094946323456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131562127 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70873181 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60688948 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8094946323456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264374735911424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8082598970659 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131572120 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1163238708015104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8095287669760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29892322 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13404753 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131572120 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131562127 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264374630826496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1õv -2…- _wandbruntime‚6518  _runtime‚6518kŽ­Ï †„ 0loss/FRt1/assignment_loss‚1.7098051265988288 0loss/FRt2/assignment_loss‚0.9085820228403265 2loss/SRqqt2/assignment_loss‚0.5122674379817869 1loss/FBt1/assignment_loss‚0.00530944561608358 epoch‚19 _step‚630 4loss/FBt2/assignment_loss‚0.00017357280134050896 /loss/FRt1/detection_loss‚0.5273314853712953 trainer/global_step‚7649 _runtime‚6518.396966418 2loss/SRqqt1/assignment_loss‚0.5424541511053632 .loss/FBt1/detection_loss‚0.322396146597079 /loss/FBt2/detection_loss‚0.5487976210335126 %loss/total_loss‚7.902843109285278 /loss/FRt2/detection_loss‚1.2778222799201666 1loss/SRqqt1/detection_loss‚0.9251382531446157 1loss/SRqqt2/detection_loss‚0.6227655662748792 % _timestamp‚1.7842793262716324e+09ò)짇‰ 2loss/SRqqt2/assignment_loss‚0.5122674379817869 1loss/FBt1/assignment_loss‚0.00530944561608358 epoch‚19 _step‚630 4loss/FBt2/assignment_loss‚0.00017357280134050896 /loss/FRt1/detection_loss‚0.5273314853712953 trainer/global_step‚7649 _runtime‚6518.396966418 2loss/SRqqt1/assignment_loss‚0.5424541511053632 .loss/FBt1/detection_loss‚0.322396146597079 /loss/FBt2/detection_loss‚0.5487976210335126 %loss/total_loss‚7.902843109285278 /loss/FRt2/detection_loss‚1.2778222799201666 1loss/SRqqt1/detection_loss‚0.9251382531446157 1loss/SRqqt2/detection_loss‚0.6227655662748792 % _timestamp‚1.7842793262716324e+09 0loss/FRt1/assignment_loss‚1.7098051265988288 0loss/FRt2/assignment_loss‚0.9085820228403265ö ³v•“ˆ‚(:ˆ ¤âçÒ­ŽË¥ gpu.0.gpu‚92.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚125.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»€‰‚(:ù ¤âçÒ¶æÙ§) disk./.usagePercent‚36.61855300322643 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚96.97299003601074& disk.nvme0n1.out‚29229.11962890625' proc.memory.availableMB‚ 495.9453125 proc.cpu.threads‚17 network.recv‚ 1.718412e+06 disk.dm-0.in‚ 317.32421875 disk.sda1.in‚0 cpu‚26.00692325598279 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 network.sent‚ 1.2164338e+07# disk.dm-0.out‚29229.11962890625 disk.nvme0n1.in‚ 317.32421875$ disk./.usageGB‚545.6429023742676 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 317.32421875! disk.rbd1.out‚100.22705078125* proc.memory.percent‚17.198920249938965 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 230.30859375 disk.sda3.out‚ 230.30859375 disk.dm-1.out‚ 230.30859375# proc.memory.rssMB‚ 2817.87109375( disk.nvme0c0n1.out‚29229.11962890625 disk.sda2.in‚0™GâÊ“Š‚(:ˆ ³âçÒê»ú£ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 É‹‚(:ö ³âçÒº¹š¦ disk.rbd0.out‚0 disk.dm-1.out‚ 230.49609375 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 318.0390625 disk.sda2.out‚0* proc.memory.percent‚17.198920249938965$ disk./.usageGB‚545.6430015563965 disk.rbd3.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚29236.9228515625 disk.sda.in‚0 disk.nvme0n1.in‚ 318.0390625 disk.sda.out‚ 230.49609375 network.sent‚ 1.2229407e+07" disk.dm-0.out‚29236.9228515625% disk.nvme0n1.out‚29236.9228515625$ memory_percent‚96.97411060333252! disk.rbd1.out‚100.28955078125 network.recv‚ 1.722619e+06 disk.dm-0.in‚ 318.0390625 disk.drbd2165.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2817.87109375* disk./.usagePercent‚36.618559659422324 disk.rbd1.in‚0.0625 cpu‚26.006312536582055 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 230.49609375 disk.dm-1.in‚0( proc.memory.availableMB‚ 495.76171875 disk.drbd2165.in‚0`‹Í+“Œ‚(:ˆ ÂâçÒ—ŒÛ¤ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚244.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dfׂ(:÷ ÂâçÒ¦Ò÷¦ disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.dm-0.in‚ 318.8671875! disk.nvme0c0n1.in‚ 318.8671875 disk.sda.in‚0 disk.sda.out‚ 230.70703125 disk.sda2.in‚0 disk.sda3.out‚ 230.70703125 disk.dm-1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚29245.53466796875# proc.memory.rssMB‚ 2817.87109375 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚26.005648236917036 proc.cpu.threads‚17 network.sent‚ 1.2232329e+07 disk.rbd3.out‚0 disk.nvme0n1.in‚ 318.8671875 disk.drbd2165.out‚0$ memory_percent‚96.97332382202148 disk.sda3.in‚0 disk.dm-1.out‚ 230.70703125# disk.dm-0.out‚29245.53466796875& disk.nvme0n1.out‚29245.53466796875 disk.rbd0.out‚0! disk.rbd1.out‚100.28955078125 disk.rbd0.in‚0& proc.memory.availableMB‚ 495.890625* proc.memory.percent‚17.198920249938965 network.recv‚ 1.725745e+06* disk./.usagePercent‚36.618556075316846$ disk./.usageGB‚545.6429481506348äÀÂ\2Ž- _wandbruntime‚6557  _runtime‚6557BÜqŸÈ $ _timestamp‚1.784279365682922e+09 _runtime‚6557.809426926 _step‚631 ' lr-AdamW/pg1‚0.00032855707899823096 ' lr-AdamW/pg2‚0.00032855707899823096 trainer/global_step‚76997Üj"ÍÇ trainer/global_step‚7699 $ _timestamp‚1.784279365682922e+09 _runtime‚6557.809426926 _step‚631 ' lr-AdamW/pg1‚0.00032855707899823096 ' lr-AdamW/pg2‚0.00032855707899823096÷|°ð2‘- _wandbruntime‚6557  _runtime‚65576NÖëó'’íO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8144970537472 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1178598506043904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61062916 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132391474 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8144629198336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚6557.809532666 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8132204403845 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71328558 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7699 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16589571 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30079832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132381495 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1178598147665408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1164432058298368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1170453518467072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1170453535506432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30079589 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8132204403845 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71328558 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7699 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16589571 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30079832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132381495 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1178598147665408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1164432058298368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.larg7Þ+ý e_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1170453518467072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1170453535506432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30079589 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1170600759496704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61070548 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257861159585792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61070548 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61070546 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132408172 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132398179 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8133556007569 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266097450633216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8133215457567 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266097555718144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16591326 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132408172 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _step‚633 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1178746759707648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8145641825280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1178746759707648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30083329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16591578 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257861178198016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8236377520128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1164578551930880 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1170600759496704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842793662945328e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚7699 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71327633 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1172711767388447 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71327633 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8145983171584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101ýòÙ'–ýM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1164578568970240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132398179 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71337624 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1170600776536064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8145641825280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8236291047424 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1178746401321984 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1170600776536064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71337624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30083584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1172712124977809 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1170600759496704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61070548 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257861159585792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61070548 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61070546 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132408172 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132398179 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8133556007569 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266097450633216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8133215457567 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266097555718144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16591326 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132408172 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _step‚633 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1178746759707648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8145641825280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1178746759707648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30083329 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16591578 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257861178198016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8236377520128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1164578551930880 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1170600759496704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842793662945328e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚7699 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71327633 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13492003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1172711767388447 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71327633 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8145983171584ù$3â2—- _wandbruntime‚6558  _runtime‚6558Ï¥7 ˜ƒ 2loss/FBt1/assignment_loss‚0.022042957830333165 /loss/FRt2/detection_loss‚1.2717477802663488 0loss/SRqqt1/detection_loss‚1.071434273531321 trainer/global_step‚7699 $ _timestamp‚1.784279366297788e+09 _runtime‚6558.423108254 _step‚634 3loss/FBt2/assignment_loss‚0.0033462878586979408 /loss/FBt1/detection_loss‚0.4533474994440601 %loss/total_loss‚8.044187005486222 0loss/FRt2/assignment_loss‚0.8530513656182881 2loss/SRqqt1/assignment_loss‚0.4021179125526131 /loss/FRt1/detection_loss‚0.5160433681836979 1loss/SRqqt2/detection_loss‚0.5879691845199326 /loss/FBt2/detection_loss‚0.7124458738566878 epoch‚19 0loss/FRt1/assignment_loss‚1.6655907138010089 2loss/SRqqt2/assignment_loss‚0.4850497880232342†CÝ™ˆ 3loss/FBt2/assignment_loss‚0.0033462878586979408 /loss/FBt1/detection_loss‚0.4533474994440601 %loss/total_loss‚8.044187005486222 0loss/FRt2/assignment_loss‚0.8530513656182881 2loss/SRqqt1/assignment_loss‚0.4021179125526131 /loss/FRt1/detection_loss‚0.5160433681836979 1loss/SRqqt2/detection_loss‚0.5879691845199326 /loss/FBt2/detection_loss‚0.7124458738566878 epoch‚19 0loss/FRt1/assignment_loss‚1.6655907138010089 2loss/SRqqt2/assignment_loss‚0.4850497880232342 2loss/FBt1/assignment_loss‚0.022042957830333165 /loss/FRt2/detection_loss‚1.2717477802663488 0loss/SRqqt1/detection_loss‚1.071434273531321 trainer/global_step‚7699 $ _timestamp‚1.784279366297788e+09 _runtime‚6558.423108254 _step‚634ú[ijs“š‚(:ˆ ÑâçÒÞÎð¤ gpu.0.gpu‚58.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚210.107' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.0214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽõÁ»þ›‚(:ó ÑâçÒ¿°º§ network.recv‚ 1.729275e+06 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 319.58203125 disk.sda2.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 319.58203125% disk.nvme0n1.out‚29252.0888671875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 231.1171875 disk.rbd3.in‚0" disk.dm-0.out‚29252.0888671875' proc.memory.availableMB‚ 495.8515625( disk./.usagePercent‚36.6185624755052 disk.sda2.out‚0 disk.sda3.out‚ 231.1171875 disk.dm-0.in‚ 319.58203125$ disk./.usageGB‚545.6430435180664 disk.rbd0.out‚0 cpu‚26.004988888070578! disk.rbd1.out‚100.35205078125 disk.rbd3.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚29252.0888671875 disk.rbd1.in‚0.0625 network.sent‚ 1.2250416e+07 disk.sda3.in‚0 disk.sda.in‚0* proc.memory.percent‚17.198920249938965 disk.sda1.in‚0 disk.sda.out‚ 231.1171875$ memory_percent‚96.97356224060059# proc.memory.rssMB‚ 2817.87109375Ý"“œ‚(:ˆ àâçÒ™·Á¤ gpu.0.gpu‚54.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶Ÿ,-þ‚(:ó àâçÒ̦֡" disk.nvme0c0n1.in‚ 320.41015625* proc.memory.percent‚17.198920249938965 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 network.recv‚ 1.733455e+06$ disk./.usageGB‚545.6430625915527 disk.sda.in‚0 disk.sda2.out‚0 cpu‚26.004381203289338 proc.cpu.threads‚17 disk.sda.out‚ 231.359375 disk.sda3.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 320.41015625# disk.dm-0.out‚29260.84130859375 disk.drbd2165.out‚0" memory_percent‚96.976637840271& disk.nvme0n1.out‚29260.84130859375( disk.nvme0c0n1.out‚29260.84130859375# proc.memory.rssMB‚ 2817.87109375 network.sent‚ 1.2302445e+07) disk./.usagePercent‚36.61856375554287 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 495.34765625! disk.rbd1.out‚100.35205078125 disk.sda2.in‚0 disk.sda3.out‚ 231.359375 disk.rbd3.out‚0 disk.dm-0.in‚ 320.41015625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 231.359375 disk.sda1.out‚0X^Ã_2ž- _wandbruntime‚6597  _runtime‚6597jºPÆŸÀ % _timestamp‚1.7842794053532987e+09 _runtime‚ 6597.47887103 _step‚635 & lr-AdamW/pg1‚0.0003268102504204639 & lr-AdamW/pg2‚0.0003268102504204639 trainer/global_step‚7749|}òãË Å & lr-AdamW/pg1‚0.0003268102504204639 & lr-AdamW/pg2‚0.0003268102504204639 trainer/global_step‚7749 % _timestamp‚1.7842794053532987e+09 _runtime‚ 6597.47887103 _step‚635û{ÿ–2¡- _wandbruntime‚6597  _runtime‚6597q H¢îO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61444516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1171757591891968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1186011253027328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1186011253027328 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1177815586988032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13577508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1177815569948672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61444514 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8182820890753 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6597.480095642 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61444514 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16694439 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259499168753664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267785854110208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1177815569948672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8286685356544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133227526 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71783010 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1179940395743361 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_splê¹ôit_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8195666039296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71783010 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267785963396608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71773033 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61444516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚7749 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1186010894648832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71773033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8286776030720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1186010894648832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1171757574852608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133217547 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30271674 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚636 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1177815586988032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13577511 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088Ôxé®ù'£óO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8195666039296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71783010 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267785963396608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71773033 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61444516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚7749 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1186010894648832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71773033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8286776030720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1186010894648832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1171757574852608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133217547 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30271674 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚636 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1177815586988032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13577511 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61444516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1171757591891968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1186011253027328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1186011253027328 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1177815586988032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13577508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1177815569948672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61444514 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8182820890753 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6597.480095642 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61444514 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16694439 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259499168753664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267785854110208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1177815569948672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8286685356544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133227526 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71783010 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1179940395743361 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976üWdt¥2¤- _wandbruntime‚6598  _runtime‚6598»£¶ºû&¥õM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1186159506691072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61452148 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8196678673408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133244224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259532597512192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚7749 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1177962828017664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 _runtime‚ 6598.09143662 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133234231 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133234231 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71792076 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1180087917468955 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8287794185216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8196337327104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30275418 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1171904085524480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8184172494477 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61452148 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267820298933248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61452146 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1177962810978304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13579253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133244224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13579253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133244224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8196678673408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133244224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259532597512192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚7749 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_ed\(›=nd/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1177962828017664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 _runtime‚ 6598.09143662 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133234231 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133234231 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71792076 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1180087917468955 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8287794185216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8196337327104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30275418 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1171904085524480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8184172494477 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61452148 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267820298933248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61452146 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1177962810978304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1185177638469632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8338006781952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8246020201984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1185177621430272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61826114 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269508613462016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261170606680064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8246020201984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1187316903413119 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8246361541120 trainer/global_step‚7799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61826114 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1193424000010752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842794452891536e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16798327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134053599 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1193424000010752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134063578 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1193423641632256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1187316545823869 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13664758 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72227485 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888±°¥_ù'µóO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1193423641632256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1187316545823869 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13664758 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72227485 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61826116 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13664761 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72227485 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8338095358976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1185177638469632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚6637.414903989 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1185177621430272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8246361541120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269508720651264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1179083125485568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72237462 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8233777927551 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8233437377661 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72237462 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30463085 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30463345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1185177638469632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8338006781952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8246020201984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1185177621430272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61826114 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269508613462016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261170606680064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8246020201984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1187316903413119 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8246361541120 trainer/global_step‚7799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61826114 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1193424000010752 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842794452891536e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16798327 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134053599 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1193424000010752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134063578 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134080276 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1193571895288832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842794459018347e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949²¯+Ä376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8247374175232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134070283 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8247032828928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261204035438592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚6638.027636838 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269543179227648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13666503 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1179229619118080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8247374175232 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1185324862459904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1187464425138825 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72246528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61833746 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8234788981385 trainer/global_step‚7799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13666506 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30466888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134080276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16800385 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61833748 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61833748 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16800638 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134070283 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72246528 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134080276 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1193571895288832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842794459018347e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8247374175232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134070283 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8247032828928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261204035438592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚6638.027636838 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269543179227648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13666503 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1179229619118080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8247374175232 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1185324862459904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1187464425138825 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72246528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61833746 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8234788981385 trainer/global_step‚7799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13666506 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30466888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134080276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16800385 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61833748 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61833748 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16800638 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134070283 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚14380¿`C2¹- _wandbruntime‚6638  _runtime‚6638@ûF¹ º„ 0loss/FRt1/assignment_loss‚1.6244605940969095 0loss/FRt2/assignment_loss‚0.8940463157624521 2loss/SRqqt2/assignment_loss‚0.5212020290077212 /loss/FRt2/detection_loss‚1.2616379805868483 1loss/SRqqt2/detection_loss‚0.6400127962304284 %loss/total_loss‚8.514193852541034 epoch‚19 trainer/global_step‚7799 .loss/FRt1/detection_loss‚0.529010271383844 /loss/FBt2/detection_loss‚1.1777478221449114 % _timestamp‚1.7842794459044771e+09 _runtime‚6638.030158139 2loss/SRqqt1/assignment_loss‚0.4887907666933604 4loss/FBt2/assignment_loss‚0.00035611661958546124 _step‚642 1loss/FBt1/assignment_loss‚0.04580770666450657 1loss/SRqqt1/detection_loss‚0.8486292903063438 /loss/FBt1/detection_loss‚0.4824921630441226Ü0ùÖ»‰ 2loss/SRqqt1/assignment_loss‚0.4887907666933604 4loss/FBt2/assignment_loss‚0.00035611661958546124 _step‚642 1loss/FBt1/assignment_loss‚0.04580770666450657 1loss/SRqqt1/detection_loss‚0.8486292903063438 /loss/FBt1/detection_loss‚0.4824921630441226 0loss/FRt1/assignment_loss‚1.6244605940969095 0loss/FRt2/assignment_loss‚0.8940463157624521 2loss/SRqqt2/assignment_loss‚0.5212020290077212 /loss/FRt2/detection_loss‚1.2616379805868483 1loss/SRqqt2/detection_loss‚0.6400127962304284 %loss/total_loss‚8.514193852541034 epoch‚19 trainer/global_step‚7799 .loss/FRt1/detection_loss‚0.529010271383844 /loss/FBt2/detection_loss‚1.1777478221449114 % _timestamp‚1.7842794459044771e+09 _runtime‚6638.030158139‚úØ]蓼‚(:ˆ œãçÒ𾛥 gpu.0.gpu‚64.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚257.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nKô½‚(:é œãçÒþ¤§ disk.sda3.out‚ 233.03125 disk.dm-1.out‚ 233.03125 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 322.66796875! disk.rbd1.out‚100.47705078125 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0# memory_percent‚96.9766616821289% proc.memory.availableMB‚ 495.34375* proc.memory.percent‚17.198920249938965 disk.dm-0.in‚ 322.66796875 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2817.87109375 disk.dm-1.in‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚29291.158203125 disk.sda.in‚0 disk.sda.out‚ 233.03125 network.recv‚ 1.746931e+06* disk./.usagePercent‚36.618518186201776$ disk./.usageGB‚545.6423835754395 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 network.sent‚ 1.2391359e+07! disk.dm-0.out‚29291.158203125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 322.66796875$ disk.nvme0n1.out‚29291.158203125 cpu‚26.001758007238905rH¥´¾‚(:’ «ãçÒÓ´Š¤ gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09žqfÿ¿‚(:ô «ãçÒŠ“ø¥ disk.dm-1.out‚ 233.265625& disk.nvme0n1.out‚29294.52001953125( disk.nvme0c0n1.out‚29294.52001953125$ memory_percent‚96.97668552398682* proc.memory.percent‚17.198920249938965 network.sent‚ 1.2440594e+07 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 322.66796875 network.recv‚ 1.751569e+06 disk.sda3.out‚ 233.265625 disk.dm-1.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚29294.52001953125 disk.rbd3.out‚0 disk.dm-0.in‚ 322.66796875 cpu‚26.00116858120354$ disk./.usageGB‚545.6403465270996 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17! disk.rbd1.out‚100.53955078125( proc.memory.availableMB‚ 495.33984375# proc.memory.rssMB‚ 2817.87109375) disk./.usagePercent‚36.61838147817849 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 322.66796875 disk.sda.in‚0 disk.sda.out‚ 233.265625 disk.sda2.out‚0 disk.sda3.in‚0T=ÒéžÀ‚(:“ ºãçÒÄÚÖ¤ gpu.0.gpu‚53.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~DÃÐþÁ‚(:ó ºãçÒØ»‚§ disk.sda2.in‚0 cpu‚26.000524877041574$ disk./.usageGB‚545.6405601501465& disk.nvme0n1.out‚29304.57958984375 disk.drbd2165.in‚0% proc.memory.availableMB‚ 495.09375 disk.dm-1.in‚0# disk.dm-0.out‚29304.57958984375 network.sent‚ 1.2447784e+07 disk.rbd3.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 323.49609375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 323.49609375# proc.memory.rssMB‚ 2817.87109375! disk.rbd1.out‚100.53955078125( disk.nvme0c0n1.out‚29304.57958984375 disk.sda3.in‚0* proc.memory.percent‚17.198920249938965* disk./.usagePercent‚36.618395814600404 disk.dm-1.out‚ 233.421875 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 233.421875 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 233.421875$ memory_percent‚96.97818756103516 network.recv‚ 1.754686e+06 disk.dm-0.in‚ 323.49609375 disk.rbd0.out‚0 disk.sda.in‚0¤vO2Â- _wandbruntime‚6677  _runtime‚6677ôiÉÃà ' lr-AdamW/pg1‚0.00032329927249282473 ' lr-AdamW/pg2‚0.00032329927249282473 trainer/global_step‚7849 % _timestamp‚1.7842794850422893e+09 _runtime‚6677.167895514 _step‚643–|KãÎÄÈ _step‚643 ' lr-AdamW/pg1‚0.00032329927249282473 ' lr-AdamW/pg2‚0.00032329927249282473 trainer/global_step‚7849 % _timestamp‚1.7842794850422893e+09 _runtime‚6677.167895514ƒ_ÑU2Å- _wandbruntime‚6677  _runtime‚6677,{é¾Å ÆíO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72691914 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271231294486016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8297057042944 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1200836388615680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62207714 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72681937 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16902592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72691914 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16902326 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8296715703808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1186408659079168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚262842044606464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8389146884608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8296715703808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8297057042944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1192539672911872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72681937 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1200836746994176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1194692695904377 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1186408642039808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 $ _timestamp‚1.784279485043568e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271231191491072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8284394414459 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1200836388615680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚262842063218688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13752011 _step‚644 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134899630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30654334 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8389231267328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7849 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62207716 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚6677.169431233 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1194693053493627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1200836746994176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1192539689951232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13752008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192539672911872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62207714 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192539689951232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0«wAø'ÇòO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62207714 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1192539689951232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72691914 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271231294486016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8297057042944 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1200836388615680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62207714 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72681937 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16902592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72691914 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16902326 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8296715703808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1186408659079168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚262842044606464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8389146884608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8296715703808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8297057042944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1192539672911872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72681937 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1200836746994176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1194692695904377 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1186408642039808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 $ _timestamp‚1.784279485043568e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271231191491072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8284394414459 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1200836388615680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚262842063218688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13752011 _step‚644 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134899630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30654334 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8389231267328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7849 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62207716 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚6677.169431233 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1194693053493627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1200836746994176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1192539689951232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13752008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192539672911872„Ý%P+2È- _wandbruntime‚6677  _runtime‚6677íëLþ&ÉøM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8297728330752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13753756 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62215346 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚6677.783012081 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1186555169751040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8285405468293 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134916328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72690989 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16904421 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842794856571717e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72690989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1192686913941504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134906335 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8285064918291 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1200985000657920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8298069677056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13753753 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1192686930980864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30658437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚7849 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134916328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8390249808896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1200984642272256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8297728330752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30658174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1186555152711680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72700980 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1192686930980864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134906335 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62215348 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62215346 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262875491977216 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134916328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72690989 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16904421 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842794856571717e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72690989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1192686913941504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134906335 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8285064918291 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1200985000657920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8298069677056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13753753 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1192686930980864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30658437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚7849 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134916328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8390249808896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1200984642272256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8297728330752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30658174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1186555152711680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72700980 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1192686930980864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134906335 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62215348 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62215346 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262875491977216 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73136389 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62589316 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264513482532864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73146366 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1193734192672768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62589314 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1208249135599104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62589316 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17005955 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1208249493977600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8440222049792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8347411205632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13839261 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272953811771904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8347752544768 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1202068845984885 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8440310626816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1199901724393472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1193734175633408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1208249493977600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8335010901367 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135725703 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1208249135599104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73136389 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62589316 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264513482532864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73146366 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1193734192672768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62589314 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1208249135599104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62589316 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17005955 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1208249493977600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8440222049792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8347411205632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13839261 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272953811771904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8347752544768 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1202068845984885 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8440310626816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1199901724393472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1193734175633408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1208249493977600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8335010901367 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135725703 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1208249135599104 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8348765178880 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62596946 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135752380 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30849047 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1193880686305280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1200048982462464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _runtime‚6717.698578007 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1208397747641344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135742387 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8441332830208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚7899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62596946 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1200048982462464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8348765178880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62596948 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272988157648896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135752380 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73155432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73145441 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17008299 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1202216725299841 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1202216367710479 % _timestamp‚1.7842795255727696e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8348423832576 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73155432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73145441 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8348423832576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62596948 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13841006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17008044 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1193880703344640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13841003 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8335681405199 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264546929903616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1208397747641344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272988262733824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8441246357504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8348765178880 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62596946 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135752380 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30849047 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1193880686305280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1200048982462464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _runtime‚6717.698578007 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1208397747641344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135742387 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8441332830208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚7899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62596946 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1200048982462464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8348765178880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62596948 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272988157648896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135752380 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73155432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73145441 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17008299 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1202216725299841 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1202216367710479 % _timestamp‚1.7842795255727696e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8348423832576 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73155432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73145441 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8348423832576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62596948 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13841006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17008044 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1193880703344640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13841003 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8335681405199 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264546929903616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1208397747641344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272988262733824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8441246357504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0‰c"ûO2Û- _wandbruntime‚6717  _runtime‚6717þtÁüÜ€ 3loss/SRqqt2/assignment_loss‚0.47112428861185274 /loss/FRt1/detection_loss‚0.5455199315590245 $ _timestamp‚1.784279525575622e+09 /loss/FRt2/detection_loss‚1.2464285563105684 .loss/FBt2/detection_loss‚1.501322845245401 _runtime‚ 6717.70104288 3loss/FBt1/assignment_loss‚0.0038496911057735255 1loss/SRqqt2/detection_loss‚0.6187250090406774 /loss/FBt1/detection_loss‚0.4491854728417351 $loss/total_loss‚8.86643371000038 epoch‚19 trainer/global_step‚7899 _step‚650 /loss/FRt1/assignment_loss‚1.713757174574463 0loss/FRt2/assignment_loss‚0.8445449286385587 1loss/FBt2/assignment_loss‚0.19285948252121063 1loss/SRqqt1/detection_loss‚0.8447806076604689 2loss/SRqqt1/assignment_loss‚0.4343357218906439ù³²¼ Ý… $ _timestamp‚1.784279525575622e+09 /loss/FRt2/detection_loss‚1.2464285563105684 .loss/FBt2/detection_loss‚1.501322845245401 _runtime‚ 6717.70104288 3loss/FBt1/assignment_loss‚0.0038496911057735255 1loss/SRqqt2/detection_loss‚0.6187250090406774 /loss/FBt1/detection_loss‚0.4491854728417351 $loss/total_loss‚8.86643371000038 epoch‚19 trainer/global_step‚7899 _step‚650 /loss/FRt1/assignment_loss‚1.713757174574463 0loss/FRt2/assignment_loss‚0.8445449286385587 1loss/FBt2/assignment_loss‚0.19285948252121063 1loss/SRqqt1/detection_loss‚0.8447806076604689 2loss/SRqqt1/assignment_loss‚0.4343357218906439 3loss/SRqqt2/assignment_loss‚0.47112428861185274 /loss/FRt1/detection_loss‚0.5455199315590245ŠT«0ô’Þ‚(:‡ çãçÒ銻¥ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:~ûß‚(:ð çãçÒ˖ߧ disk.dm-1.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 325.75390625# proc.memory.rssMB‚ 2817.87109375 network.recv‚ 1.76599e+06 disk.sda2.out‚0& disk.nvme0n1.out‚29328.16748046875( disk.nvme0c0n1.out‚29328.16748046875 network.sent‚ 1.2531774e+07 disk.sda.in‚0 disk.sda3.out‚ 234.21875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 325.75390625 disk.rbd0.out‚0 disk.sda2.in‚0$ memory_percent‚96.97847366333008$ disk./.usageGB‚545.6424942016602 disk.rbd1.in‚0.0625 disk.rbd0.in‚0 cpu‚25.998630947500747 disk.sda.out‚ 234.21875 disk.drbd2165.in‚0* disk./.usagePercent‚36.618525610420264# disk.dm-0.out‚29328.16748046875 disk.sda1.in‚0* proc.memory.percent‚17.198920249938965 disk.sda3.in‚0 disk.dm-1.out‚ 234.21875! disk.rbd1.out‚100.66455078125 disk.nvme0n1.in‚ 325.75390625 disk.sda1.out‚0& proc.memory.availableMB‚ 495.046875 proc.cpu.threads‚17ªÈ‘à‚(:† öãçÒý†€¥ gpu.0.gpu‚74.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ©Sá‚(:ú öãçÒ㨂§ disk.rbd3.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚29333.78564453125 disk.sda.in‚0 disk.sda.out‚ 234.35546875 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 325.75390625 network.sent‚ 1.2583835e+07 disk.dm-1.out‚ 234.35546875 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd1.out‚100.66455078125& disk.nvme0n1.out‚29333.78564453125 disk.drbd2165.in‚0 network.recv‚ 1.770279e+06 disk.rbd1.in‚0.0625 cpu‚25.998012974842208) disk./.usagePercent‚36.61818563241479 disk.sda3.out‚ 234.35546875 disk.dm-1.in‚0' proc.memory.availableMB‚ 495.0234375* proc.memory.percent‚17.198920249938965 proc.cpu.threads‚17$ disk./.usageGB‚545.6374282836914 disk.sda2.out‚0 disk.dm-0.in‚ 325.75390625$ memory_percent‚96.97861671447754 disk.sda2.in‚0# disk.dm-0.out‚29333.78564453125 disk.nvme0n1.in‚ 325.75390625# proc.memory.rssMB‚ 2817.87109375w‰!‘â‚(:† …äçÒ¤˜ý£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ5ò¤ã‚(:÷ …äçÒ¹ð‡¦ disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 cpu‚25.997399701263507 proc.cpu.threads‚17 disk.rbd3.out‚0" disk.dm-0.out‚29340.8681640625% disk.nvme0n1.out‚29340.8681640625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 326.58203125) disk./.usagePercent‚36.61854250691753 disk.rbd1.in‚0.0625' disk.nvme0c0n1.out‚29340.8681640625 disk.sda.out‚ 234.79296875 disk.sda2.in‚0$ memory_percent‚96.97861671447754 disk.sda3.out‚ 234.79296875 disk.dm-1.in‚0 network.recv‚ 1.773665e+06 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 495.0234375# proc.memory.rssMB‚ 2817.87109375 network.sent‚ 1.2588941e+07! disk.rbd1.out‚100.72705078125* proc.memory.percent‚17.198920249938965$ disk./.usageGB‚545.6427459716797 disk.drbd2165.in‚0 disk.dm-0.in‚ 326.58203125 disk.rbd0.out‚0 disk.dm-1.out‚ 234.79296875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 326.58203125ÿ-Rð2ä- _wandbruntime‚6757  _runtime‚6757•Ž% ÇåÁ & lr-AdamW/pg1‚0.0003197659524836417 & lr-AdamW/pg2‚0.0003197659524836417 trainer/global_step‚7949 % _timestamp‚1.7842795649243464e+09 _runtime‚6757.049925413 _step‚651&{vîÌæÆ & lr-AdamW/pg2‚0.0003197659524836417 trainer/global_step‚7949 % _timestamp‚1.7842795649243464e+09 _runtime‚6757.049925413 _step‚651 & lr-AdamW/pg1‚0.0003197659524836417‹‹Z˺2ç- _wandbruntime‚6757  _runtime‚6757ôkvô'èîO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1215661882582528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62970916 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17110538 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1209444996065393 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7842795649254122e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73600818 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1215661882582528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274676313775616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266184920459264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209445353654643 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8385627388275 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8398448046592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1215662240961024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1207263792914432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚6757.051157359 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31036783 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1201059726266368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136561755 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8398106707456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1201059709227008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73590841 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8398448046592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73590841 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274676210780672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8398106707456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1215662240961024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1207263792914432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚6757.051157359 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31036783 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1201059726266368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136561755 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8398106707456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1201059709227008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73590841 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8398448046592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.svÖ0ñ·mall_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73590841 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274676210780672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8398106707456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62970916 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17110538 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1209444996065393 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7842795649254122e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73600818 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1215661882582528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274676313775616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266184920459264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209445353654643 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8385627388275 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8398448046592Œ ®‡Ä2ê- _wandbruntime‚6757  _runtime‚6757w†þ&ëøM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1207411033944064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136588432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8399460680704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8399460680704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842795655382013e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8386297892107 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266218367830016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1207411016904704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136588432 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1215810494624768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31040876 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1201206219898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8386638442109 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73609884 trainer/global_step‚7949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1215810136239104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31040611 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚653 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136578439 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62978546 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209592875380349 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1215810494624768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73599893 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266218349217792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62978548 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209592517790987 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1201206236938240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73599893 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17112620 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8492301509632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274710762103808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8399119334400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13928256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1207411033944064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136588432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8399460680704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8399460680704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842795655382013e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8386297892107 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266218367830016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1207411016904704 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136588432 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1215810494624768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31040876 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1201206219898880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8386638442109 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73609884 trainer/global_step‚7949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1215810136239104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31040611 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚653 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136578439 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62978546 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209592875380349 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1215810494624768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73599893 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266218349217792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62978548 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209592517790987 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1201206236938240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73599893 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17112620 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8492301509632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274710762103808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8399119334400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13928256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QL-ô'úîO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14013761 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚7999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63352516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267856376997888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1223074987944448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8449143548416 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17214012 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223074629565952 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8448802209280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276398708749824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31228031 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1214625844396032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17214270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8448802209280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8542350364160 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1216821146145901 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8435903325293 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1214625827356672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74055270 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74055270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14013758 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1214625844396032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1216821503735151 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63352516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31227770 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276398811744768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8436243875183 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137397807 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63352514b&À’ù'ûóO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8448802209280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276398708749824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31228031 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1214625844396032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17214270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8448802209280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8542350364160 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1216821146145901 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8435903325293 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1214625827356672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74055270 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74055270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14013758 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1214625844396032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1216821503735151 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63352516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31227770 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276398811744768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8436243875183 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137397807 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63352514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14013761 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚7999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63352516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267856376997888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1223074987944448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8449143548416 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17214012 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223074629565952 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8450156182528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137424484 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74054345 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31231591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1223223241608192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63360148 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8543379304448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63360148 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 % _timestamp‚1.7842796053244307e+09 _step‚657 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74064336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276433166448640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1208531753492480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63360146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31231833 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137414491 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8437254929017 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137414491 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1208531770531840 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8436914379015 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1223222883222528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1216969025460857 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137424484 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137424484 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74054345 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31231591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1223223241608192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63360148 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8543379304448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63360148 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 % _timestamp‚1.7842796053244307e+09 _step‚657 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74064336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276433166448640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMon/2‚itor.on_train_batch_end/requested_bytes.large_pool.freed‚1208531753492480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63360146 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31231833 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137414491 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8437254929017 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137414491 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1208531770531840 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8436914379015 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1223222883222528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1216969025460857 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137424484 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304‘)©-2ÿ- _wandbruntime‚6797  _runtime‚6797¼÷& €ƒ 3loss/FBt2/assignment_loss‚0.0005982328848465484 /loss/FBt1/detection_loss‚0.4267798248840958 _step‚658 2loss/SRqqt2/assignment_loss‚0.5020889290129558 trainer/global_step‚7999 2loss/SRqqt1/assignment_loss‚0.5516579426904008 /loss/FRt2/detection_loss‚1.2061574046068184 1loss/SRqqt2/detection_loss‚0.6213066634173239 %loss/total_loss‚7.770377865524482 epoch‚19 _runtime‚6797.452579015 /loss/FRt1/assignment_loss‚1.630521251378875 0loss/FRt2/assignment_loss‚0.8021967401051654 1loss/FBt1/assignment_loss‚0.01058061962127951 /loss/FRt1/detection_loss‚0.5340395056701118 1loss/SRqqt1/detection_loss‚0.8274772912798236 /loss/FBt2/detection_loss‚0.6569734599727851 % _timestamp‚1.7842796053272464e+09X¢˜Nˆ /loss/FRt1/detection_loss‚0.5340395056701118 1loss/SRqqt1/detection_loss‚0.8274772912798236 /loss/FBt2/detection_loss‚0.6569734599727851 % _timestamp‚1.7842796053272464e+09 3loss/FBt2/assignment_loss‚0.0005982328848465484 /loss/FBt1/detection_loss‚0.4267798248840958 _step‚658 2loss/SRqqt2/assignment_loss‚0.5020889290129558 trainer/global_step‚7999 2loss/SRqqt1/assignment_loss‚0.5516579426904008 /loss/FRt2/detection_loss‚1.2061574046068184 1loss/SRqqt2/detection_loss‚0.6213066634173239 %loss/total_loss‚7.770377865524482 epoch‚19 _runtime‚6797.452579015 /loss/FRt1/assignment_loss‚1.630521251378875 0loss/FRt2/assignment_loss‚0.8021967401051654 1loss/FBt1/assignment_loss‚0.01058061962127951’þIÖF‚‚(:’ ÁäçÒ‡ƒÊ¤ gpu.0.gpu‚20.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚107.865' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.572999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶Íýƒ‚(:ò ÁäçÒÁ•Ì¦ disk.dm-0.out‚29370.43359375" disk.nvme0c0n1.in‚ 328.83984375 disk.sda.out‚ 235.86328125 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2817.87890625 disk.drbd2165.out‚0 network.recv‚ 1.788106e+06* disk./.usagePercent‚36.618532522623696$ disk./.usageGB‚545.6425971984863 disk.dm-1.out‚ 235.86328125! disk.rbd1.out‚100.91455078125* proc.memory.percent‚17.198967933654785 disk.rbd3.out‚0# disk.nvme0n1.out‚29370.43359375 cpu‚26.001940088653615 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 328.83984375 disk.rbd0.out‚0% disk.nvme0c0n1.out‚29370.43359375$ memory_percent‚96.97813987731934' proc.memory.availableMB‚ 495.1015625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 328.83984375 proc.cpu.threads‚17 network.sent‚ 1.2677948e+07 disk.sda2.out‚0 disk.sda3.out‚ 235.86328125 disk.rbd3.in‚0 disk.rbd0.in‚0†r¿2„- _wandbruntime‚6814  _runtime‚6814t×?’b=…Üz ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_3‚1 particle/accuracy_5_of_5‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975057244300842 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6282785534858704 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5185595750808716 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8642241954803467 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7397181987762451 ,jet/accuracy_4_of_4‚0.013899499550461769 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27272871136665344 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7412858009338379 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25718924403190613 'particle/f_score‚0.6386162638664246 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969390630722046 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6471315026283264 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1817644089460373 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03705437481403351 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2724670171737671 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.359375 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24290557205677032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7434164881706238 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04558613523840904 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6711965799331665 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6043750047683716 *jet/accuracy_2_of_3‚0.5912100672721863 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7248234152793884 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944258332252502 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6463420391082764 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14243657886981964 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.34947219491004944 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7688204646110535 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26405155658721924 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7298609614372253 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999627947807312 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.817460298538208 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07417788356542587 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5056257247924805 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.260054349899292 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚19 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.35923048853874207 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44980281591415405 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9328016638755798 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.13197357952594757 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8326908946037292 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06860390305519104 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5917795896530151 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5458258986473083 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.65133136510849 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6550342440605164 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6791723370552063 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8581246137619019 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7292510271072388 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0570409819483757 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16637006402015686 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03524206578731537 *jet/accuracy_1_of_2‚0.8154844641685486 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.787905216217041 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7890009880065918 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27202799916267395 trainer/global_step‚7999 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7297567129135132 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 % _timestamp‚1.7842796226114624e+09 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3919905722141266 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01952046900987625 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29443782567977905 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6677530407905579 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6580666899681091 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5302964448928833 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9343613982200623 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7217976450920105 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19498904049396515 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9968493580818176 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6580666899681091 /particle/accuracy_3_of_4‚0.9940476417541504 _runtime‚6814.737822082 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6888600587844849 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7888081669807434 (particle/accuracy‚0.8476911187171936 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07394019514322281 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7287392616271973 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _step‚659 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5603966116905212 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7127688527107239 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8445956707000732 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3360747694969177 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9081200957298279 particle/accuracy_3_of_5‚1 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.70667964220047 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031075432896614075 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6067691445350647 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8020408153533936 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3096129894256592 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35628271102905273 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23841935396194458 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3598487377166748 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3505789041519165 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7623545527458191 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7595049738883972 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36020466685295105 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7537441849708557 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39412498474121094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8581246137619019 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30115312337875366 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6869105100631714 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5013526678085327 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 +jet/accuracy_2_of_2‚0.16939298808574677 *jet/accuracy_2_of_4‚0.7970267534255981 *jet/accuracy_3_of_4‚0.0867510586977005 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032185815274715424 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38799890875816345 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994588494300842 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.34791845083236694 *jet/accuracy_1_of_1‚0.3774218261241913 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7032856345176697 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29065459966659546 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9099339246749878 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 )loss/val_total_loss‚8.014956323847137 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891303777694702 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3505789041519165 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 *jet/accuracy_1_of_4‚0.9786158800125122 particle/accuracy_1_of_4‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9523415565490723 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3047701418399811 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7817582488059998 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8726603388786316 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969701170921326 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01952046900987625 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39939671754837036 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6722061634063721 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6669967174530029 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6674081087112427 +particle/sensitivity‚0.6267241835594177 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9977399110794067 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6880141496658325 6validation_average_jet_accuracy‚0.4120032489299774 jet/accuracy_2_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8333712220191956 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.543749988079071 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17947262525558472 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34118035435676575 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.1466666758060455371£È†áz 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7292510271072388 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0570409819483757 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16637006402015686 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03524206578731537 *jet/accuracy_1_of_2‚0.8154844641685486 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.787905216217041 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7890009880065918 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27202799916267395 trainer/global_step‚7999 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7297567129135132 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 % _timestamp‚1.7842796226114624e+09 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3919905722141266 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01952046900987625 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29443782567977905 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6677530407905579 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6580666899681091 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5302964448928833 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9343613982200623 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7217976450920105 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19498904049396515 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9968493580818176 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6580666899681091 /particle/accuracy_3_of_4‚0.9940476417541504 _runtime‚6814.737822082 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6888600587844849 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7888081669807434 (particle/accuracy‚0.8476911187171936 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07394019514322281 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7287392616271973 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 _step‚659 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5603966116905212 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7127688527107239 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8445956707000732 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3360747694969177 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9081200957298279 particle/accuracy_3_of_5‚1 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.70667964220047 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031075432896614075 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6067691445350647 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8020408153533936 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3096129894256592 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.35628271102905273 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23841935396194458 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3598487377166748 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3505789041519165 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7623545527458191 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7595049738883972 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.36020466685295105 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7537441849708557 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39412498474121094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8581246137619019 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30115312337875366 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6869105100631714 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5013526678085327 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 +jet/accuracy_2_of_2‚0.16939298808574677 *jet/accuracy_2_of_4‚0.7970267534255981 *jet/accuracy_3_of_4‚0.0867510586977005 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032185815274715424 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.38799890875816345 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994588494300842 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.34791845083236694 *jet/accuracy_1_of_1‚0.3774218261241913 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7032856345176697 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.29065459966659546 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9099339246749878 > «Ÿ%&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 )loss/val_total_loss‚8.014956323847137 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891303777694702 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3505789041519165 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 *jet/accuracy_1_of_4‚0.9786158800125122 particle/accuracy_1_of_4‚1 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9523415565490723 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3047701418399811 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7817582488059998 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8726603388786316 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969701170921326 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01952046900987625 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39939671754837036 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6722061634063721 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6669967174530029 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6674081087112427 +particle/sensitivity‚0.6267241835594177 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9977399110794067 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6880141496658325 6validation_average_jet_accuracy‚0.4120032489299774 jet/accuracy_2_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8333712220191956 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.543749988079071 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17947262525558472 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_5‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.34118035435676575 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14666667580604553 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_3‚1 particle/accuracy_5_of_5‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975057244300842 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6282785534858704 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5185595750808716 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8642241954803467 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7397181987762451 ,jet/accuracy_4_of_4‚0.013899499550461769 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.27272871136665344 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7412858009338379 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25718924403190613 'particle/f_score‚0.6386162638664246 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969390630722046 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6471315026283264 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1817644089460373 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03705437481403351 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2724670171737671 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.359375 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24290557205677032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7434164881706238 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04558613523840904 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6711965799331665 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6043750047683716 *jet/accuracy_2_of_3‚0.5912100672721863 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7248234152793884 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944258332252502 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6463420391082764 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14243657886981964 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.34947219491004944 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7688204646110535 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26405155658721924 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7298609614372253 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999627947807312 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.817460298538208 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07417788356542587 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5056257247924805 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.260054349899292 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚19 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.35923048853874207 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44980281591415405 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9328016638755798 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.13197357952594757 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8326908946037292 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06860390305519104 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5917795896530151 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5458258986473083 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.65133136510849 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6550342440605164 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6791723370552063 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8581246137619019 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1“Én§Mž‡‚(:“ ÐäçÒÓõ¶¥ gpu.0.gpu‚65.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œoh¼÷ˆ‚(:ì ÐäçÒòæƧ! disk.rbd1.out‚100.91455078125# disk.nvme0n1.out‚29626.66796875$ disk./.usageGB‚545.6473083496094" disk.nvme0c0n1.in‚ 329.66796875 disk.nvme0n1.in‚ 329.66796875( proc.memory.availableMB‚ 730.00390625 disk.sda2.out‚0 disk.dm-0.out‚29626.66796875 disk.sda2.in‚0 disk.dm-0.in‚ 329.66796875 disk.rbd0.out‚0 cpu‚26.00342013086652 disk.sda.in‚0 disk.sda.out‚ 236.0546875 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2689.06640625 proc.cpu.threads‚17) disk./.usagePercent‚36.61884869192848% disk.nvme0c0n1.out‚29626.66796875 disk.dm-1.out‚ 236.0546875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.412758827209473 network.sent‚ 1.27279e+07 network.recv‚ 1.792447e+06 disk.sda3.out‚ 236.0546875 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚95.54440975189209+;䓉‚(:ˆ ßäçÒª¨é¤ gpu.0.gpu‚92.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝK ¹þŠ‚(:ó ßäçҲঠnetwork.sent‚ 1.2787562e+07 disk.nvme0n1.in‚ 330.3828125 disk.rbd0.in‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 330.3828125" disk.dm-0.out‚29883.6240234375$ memory_percent‚95.54305076599121# proc.memory.rssMB‚ 2689.06640625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0! disk.nvme0c0n1.in‚ 330.3828125* proc.memory.percent‚16.412758827209473% disk.nvme0n1.out‚29883.6240234375 disk.sda2.in‚0 disk.sda3.out‚ 236.40234375 disk.dm-1.out‚ 236.40234375$ disk./.usageGB‚545.6464500427246' disk.nvme0c0n1.out‚29883.6240234375 disk.sda.out‚ 236.40234375! disk.rbd1.out‚100.97705078125 disk.drbd2165.out‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 730.2265625 cpu‚26.00280167369836 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 1.797152e+06) disk./.usagePercent‚36.61879109023328 disk.sda1.in‚0 disk.drbd2165.in‚0&¤^f“‹‚(:ˆ îäçÒóÞŒ¥ gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì`ÔýŒ‚(:ò îäçÒû¹“§' proc.memory.availableMB‚ 730.2265625 disk.sda3.out‚ 238.28515625" disk.dm-0.out‚29900.5595703125# proc.memory.rssMB‚ 2689.06640625 network.sent‚ 1.2792667e+07 disk.sda2.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 network.recv‚ 1.800376e+06 disk.sda3.in‚0! disk.rbd1.out‚100.97705078125 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 331.2109375' disk.nvme0c0n1.out‚29900.5595703125$ memory_percent‚95.54305076599121 disk.rbd3.in‚0% disk.nvme0n1.out‚29900.5595703125 disk.sda1.in‚0 disk.rbd0.out‚0 cpu‚26.0022127958176 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 238.28515625* proc.memory.percent‚16.412758827209473 disk.dm-0.in‚ 331.2109375 disk.drbd2165.out‚0) disk./.usagePercent‚36.61878699411273 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.sda.in‚0$ disk./.usageGB‚545.6463890075684 disk.dm-1.out‚ 238.28515625 disk.nvme0n1.in‚ 331.2109375˜Ã2- _wandbruntime‚6856  _runtime‚6856gUÝÇŽÁ & lr-AdamW/pg2‚0.0003162111981325055 trainer/global_step‚8049 % _timestamp‚1.7842796640750623e+09 _runtime‚6856.200676452 _step‚660 & lr-AdamW/pg1‚0.0003162111981325055K»Ù‡ÌÆ _step‚660 & lr-AdamW/pg1‚0.0003162111981325055 & lr-AdamW/pg2‚0.0003162111981325055 trainer/global_step‚8049 % _timestamp‚1.7842796640750623e+09 _runtime‚6856.200676452”ð:k%2- _wandbruntime‚6856  _runtime‚6856¶‰ò'‘ìO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74570072 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1223917462528000 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1217619456667648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63854780 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74580049 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1217619439628288 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74580049 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 6856.20536378 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1232435034786304 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1223917462528000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17353727 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269543727024128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8504239537435 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31457255 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1232434676407808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63854780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31457518 $ _timestamp‚1.784279664076113e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63854782 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74570072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1232434676407808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8517230919168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚661 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138424852 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8611760294912 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8504580087325 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8517572258304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269543708411904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8504580087325 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8517572258304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269543708411904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74580049 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1217619439628288 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74580049 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 6856.20536378 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1232435034786304 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1223917462528000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17353727 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269543727024128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8504239537435 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2X«Ãf LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31457255 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1232434676407808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63854780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31457518 $ _timestamp‚1.784279664076113e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63854782 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74570072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1232434676407808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0•„Mòo2“- _wandbruntime‚6856  _runtime‚6856é:‚ þ&”øM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1232582930064384 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1226271558480679 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74579124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1224064686518272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1224064686518272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269577155782656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8518243546112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1226271200891317 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138451529 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17355808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚6856.815265551 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1224064703557632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269577137170432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138451529 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278190022653440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8505591141159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138441536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14105273 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1232583288450048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74589115 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63862414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8612778300928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17356083 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚8049 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31461359 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14105276 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8612866870784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31461081 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚662 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8518584892416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1232583288450048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74589115 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63862414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8612778300928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17356083 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚8049 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31461359 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14105276 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8612866870784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31461081 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚662 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8518584892416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1224064686518272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1224064686518272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269577155782656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8518243546112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1226271200891317 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138451529 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17355808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚6856.815265551 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1224064703557632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269577137170432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138451529 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278190022653440–ÔY°2–- _wandbruntime‚6856  _runtime‚6856%„÷H—û /loss/FBt2/detection_loss‚0.5539074396534313 epoch‚20 _runtime‚6856.817082039 0loss/FRt2/assignment_loss‚0.7882273575311066 /loss/FBt1/assignment_loss‚0.083514816167219 0loss/SRqqt2/detection_loss‚0.586386462540207 %loss/total_loss‚7.689884180877417 0loss/FRt1/assignment_loss‚1.5419109966975963 /loss/FRt1/detection_loss‚0.5369290806303285 trainer/global_step‚8049 3loss/SRqqt1/assignment_loss‚0.48690103579797195 1loss/FBt2/assignment_loss‚0.05769215601356225 1loss/SRqqt1/detection_loss‚0.8456442437836003 .loss/FBt1/detection_loss‚0.526400513347104 $ _timestamp‚1.784279664691681e+09 _step‚663 0loss/SRqqt2/assignment_loss‚0.48569953479869 .loss/FRt2/detection_loss‚1.196670543916601Å(˜b˜€ trainer/global_step‚8049 3loss/SRqqt1/assignment_loss‚0.48690103579797195 1loss/FBt2/assignment_loss‚0.05769215601356225 1loss/SRqqt1/detection_loss‚0.8456442437836003 .loss/FBt1/detection_loss‚0.526400513347104 $ _timestamp‚1.784279664691681e+09 _step‚663 0loss/SRqqt2/assignment_loss‚0.48569953479869 .loss/FRt2/detection_loss‚1.196670543916601 /loss/FBt2/detection_loss‚0.5539074396534313 epoch‚20 _runtime‚6856.817082039 0loss/FRt2/assignment_loss‚0.7882273575311066 /loss/FBt1/assignment_loss‚0.083514816167219 0loss/SRqqt2/detection_loss‚0.586386462540207 %loss/total_loss‚7.689884180877417 0loss/FRt1/assignment_loss‚1.5419109966975963 /loss/FRt1/detection_loss‚0.5369290806303285—Vfyž™‚(:“ ýäçÒƒãᥠgpu.0.gpu‚76.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.198' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.439600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0… î~úš‚(:ï ýäçÒúÇ…¨* disk./.usagePercent‚36.618792370270945 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚29908.185546875 network.sent‚ 1.2808873e+07 disk.nvme0n1.in‚ 331.2109375 disk.sda1.out‚0 disk.sda3.out‚ 240.6953125 disk.dm-1.in‚0 disk.rbd3.in‚0 network.recv‚ 1.803714e+06 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 240.6953125$ disk.nvme0n1.out‚29908.185546875 disk.sda.out‚ 240.6953125 disk.sda2.out‚0# proc.memory.rssMB‚ 2689.06640625 cpu‚26.001586475543192 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚95.54610252380371' proc.memory.availableMB‚ 729.7265625& disk.nvme0c0n1.out‚29908.185546875 disk.rbd1.in‚0.0625! disk.rbd1.out‚101.03955078125 disk.dm-0.in‚ 331.2109375* proc.memory.percent‚16.412758827209473$ disk./.usageGB‚545.6464691162109! disk.nvme0c0n1.in‚ 331.2109375 disk.sda.in‚0sÌÍ›‚(:’ ŒåçÒ®ëé¤ gpu.0.gpu‚77.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aö윂(:÷ ŒåçÒÀ«”§( proc.memory.availableMB‚ 730.19140625 disk.drbd2165.in‚0 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 331.92578125! disk.rbd1.out‚101.03955078125 disk.dm-0.in‚ 331.92578125# memory_percent‚95.5432653427124 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2689.06640625 network.sent‚ 1.2860803e+07) disk./.usagePercent‚36.61878776213533 disk.sda1.in‚0# disk.dm-0.out‚29919.76025390625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0* proc.memory.percent‚16.412758827209473& disk.nvme0n1.out‚29919.76025390625" disk.nvme0c0n1.in‚ 331.92578125 disk.sda.out‚ 241.2265625 disk.dm-1.in‚0 cpu‚26.000976763437123$ disk./.usageGB‚545.6464004516602 disk.sda1.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚29919.76025390625 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 241.2265625 disk.dm-1.out‚ 241.2265625 network.recv‚ 1.807738e+06Ã9ö2- _wandbruntime‚6895  _runtime‚6895¸ zɞà _runtime‚6895.995969007 _step‚664 ' lr-AdamW/pg1‚0.00031442606825946115 ' lr-AdamW/pg2‚0.00031442606825946115 trainer/global_step‚8099 % _timestamp‚1.7842797038694994e+09@¥dÎŸÈ ' lr-AdamW/pg1‚0.00031442606825946115 ' lr-AdamW/pg2‚0.00031442606825946115 trainer/global_step‚8099 % _timestamp‚1.7842797038694994e+09 _runtime‚6895.995969007 _step‚664˜‡Äú2 - _wandbruntime‚6895  _runtime‚6895¼fš™b¡ìO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139270883 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8568267760128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17458179 trainer/global_step‚8099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64236382 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64236380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784279703870256e+09 _runtime‚6895.996068235 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1239847423391232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1233499829246231 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8567926420992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279878375112192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75034501 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31648960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14190781 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8663210161664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64236380 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1239847781769728 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1239847423391232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139260904 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1231279514009600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8663129973248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1231279514009600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1231279496970240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64236382 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75034501 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555196574233 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oäãtÇversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271215146338304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554856024343 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1239847781769728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1224944973221888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1224944990261248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75024524 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8568267760128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31648697 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0yš4 ÷'¢ñO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271215146338304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554856024343 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1239847781769728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1224944973221888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1224944990261248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75024524 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8568267760128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31648697 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139270883 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8568267760128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17458179 trainer/global_step‚8099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64236382 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64236380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784279703870256e+09 _runtime‚6895.996068235 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1239847423391232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1233499829246231 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8567926420992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279878375112192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75034501 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31648960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14190781 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8663210161664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64236380 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1239847781769728 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1239847423391232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139260904 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1231279514009600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8663129973248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1231279514009600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1231279496970240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64236382 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75034501 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555196574233 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088™ƒmý†2£- _wandbruntime‚6896  _runtime‚6896ÚÖ[qý&¤÷M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271248575096832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64244012 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1231426755039232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64244012 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271248593709056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚666 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139287581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14192523 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17460259 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚6896.607186806 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75033576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8664243096576 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1233647708561187 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1239996035433472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8555867078065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1225091483893760 % _timestamp‚1.7842797044814377e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8569280394240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8664167109632 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1225091500933120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75033576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1231426737999872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31652532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279912836805632 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1233647350971825 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8568939047936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64244014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1239995677047808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139287581 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17460009 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚8099 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8556207628067 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139277588 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139277588 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269b»q’'¥üM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1225091483893760 % _timestamp‚1.7842797044814377e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8569280394240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8664167109632 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1225091500933120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75033576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1231426737999872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31652532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279912836805632 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1233647350971825 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8568939047936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64244014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1239995677047808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139287581 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17460009 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚8099 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8556207628067 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139277588 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139277588 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271248575096832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64244012 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1231426755039232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64244012 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271248593709056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚666 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139287581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14192523 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17460259 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚6896.607186806 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75033576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8664243096576 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1233647708561187 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDevnÌÆíÛiceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1239996035433472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8555867078065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14278028 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8714667006464 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1247260170374656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8618621922816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75488953 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1232270506815488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1232270523854848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140096956 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281601181568512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8714597303808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281601269883392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1238641548451840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872ÚcÖ÷'´ñO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14278028 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8714667006464 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1247260170374656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8618621922816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75488953 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1232270506815488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1232270523854848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140096956 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281601181568512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8714597303808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281601269883392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1238641548451840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64617982 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269`±C¦2µ- _wandbruntime‚6936  _runtime‚6936,˧ýŒ ¶øM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1241023858641695 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1241023501052333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272920031635456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8619975896064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17564118 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14279776 _runtime‚6936.624040379 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17564375 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1238788806520832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281635630989312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1238788789481472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64625612 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75488028 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1238788806520832 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1247408782416896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31843891 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1247408782416896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140113640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31844151 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatŸ¹õorsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75498019 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75488028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8606824114975 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64625614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚670 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140113640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8715617966080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1232417034526720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842797444982607e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1247408424031232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64625612 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140123633 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140123633 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75498019 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1238788789481472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8715700244480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8606483564973 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8619975896064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17564118 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14279776 _runtime‚6936.624040379 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17564375 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1238788806520832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281635630989312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1238788789481472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64625612 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75488028 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1238788806520832 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1247408782416896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31843891 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1247408782416896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140113640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31844151 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75498019 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75488028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8606824114975 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64625614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚670 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140113640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8715617966080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1232417034526720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842797444982607e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1247408424031232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64625612 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140123633 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140123633 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75498019 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1238788789481472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8715700244480 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8606483564973 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1239596040409088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1246003616972800 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1254672917358080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274558040803328 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1254673275736576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8669317424640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚673 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17666497 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1254672917358080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8765993491968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1239596057448448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1239596040409088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1246003616972800 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1254672917358080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274558040803328 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1254673275736576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8669317424640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚673 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17666497 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1254672917358080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8765993491968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0¡™:ªz2Ç- _wandbruntime‚6977  _runtime‚6977öz"Ÿþ&ÈøM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283358568359936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8657440601883 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1254821529400320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1254821171014656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1246150840963072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32035296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283358456983552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1246150858002432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1246150858002432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1246150840963072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140949692 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17668552 trainer/global_step‚8199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65007212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14367023 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚6977.191507003 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75942480 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8670671397888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8670671397888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75942480 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14367026 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8670330051584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1248399651132841 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1239742568120320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75952471 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32035578 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140959685 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65007214 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17668273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842797850657456e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8767006033920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚02~»Ì'ÉýM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75952471 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32035578 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140959685 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65007214 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17668273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842797850657456e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8767006033920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283358568359936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8657440601883 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1254821529400320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1254821171014656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1246150840963072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32035296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283358456983552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1246150858002432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1246150858002432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1246150840963072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140949692 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17668552 trainer/global_step‚8199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65007212 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14367023 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚6977.191507003 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75942480 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8670671397888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8670671397888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75942480 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14367026 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8670330051584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1248399651132841 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1239742568120320¢Îæ2Ê- _wandbruntime‚6977  _runtime‚6977Ÿ_ ˆ epoch‚20 trainer/global_step‚8199 $ _timestamp‚1.784279785068762e+09 _runtime‚6977.194088106 _step‚675 0loss/FRt1/assignment_loss‚1.5836697444482757 2loss/SRqqt1/assignment_loss‚0.4009244396095451 3loss/FBt2/assignment_loss‚8.702044859774106e-05 0loss/FRt1/detection_loss‚0.49688667851930945 1loss/SRqqt1/detection_loss‚0.8638317360189455 0loss/FRt2/assignment_loss‚0.9047949478002142 1loss/SRqqt2/detection_loss‚0.5736266977161499 /loss/FBt1/detection_loss‚0.3327596621449165 /loss/FBt2/detection_loss‚0.6323064186377451 3loss/SRqqt2/assignment_loss‚0.46473721833071197 2loss/FBt1/assignment_loss‚0.026916146334009183 /loss/FRt2/detection_loss‚1.2613823921246483 %loss/total_loss‚7.541923102133069ã5êÌ‹ 0loss/FRt1/assignment_loss‚1.5836697444482757 2loss/SRqqt1/assignment_loss‚0.4009244396095451 3loss/FBt2/assignment_loss‚8.702044859774106e-05 0loss/FRt1/detection_loss‚0.49688667851930945 1loss/SRqqt1/detection_loss‚0.8638317360189455 0loss/FRt2/assignment_loss‚0.9047949478002142 1loss/SRqqt2/detection_loss‚0.5736266977161499 /loss/FBt1/detection_loss‚0.3327596621449165 /loss/FBt2/detection_loss‚0.6323064186377451 3loss/SRqqt2/assignment_loss‚0.46473721833071197 2loss/FBt1/assignment_loss‚0.026916146334009183 /loss/FRt2/detection_loss‚1.2613823921246483 %loss/total_loss‚7.541923102133069 epoch‚20 trainer/global_step‚8199 $ _timestamp‚1.784279785068762e+09 _runtime‚6977.194088106 _step‚675£ŸÙœŽnÍ‚(:† õåçÒßÚº¥ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0#ÎX2# gpu.0.uncorrectedMemoryErrors‚0Чˆúù΂(:î õåçÒŸñò§ network.recv‚ 1.833226e+06 disk.sda1.in‚0& disk.nvme0c0n1.out‚30014.384765625 disk.dm-1.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2689.06640625$ disk.nvme0n1.out‚30014.384765625 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.rbd1.out‚101.35205078125 disk.rbd3.in‚0 network.sent‚ 1.3022428e+07) disk./.usagePercent‚36.62066404135378$ disk./.usageGB‚545.6743583679199 disk.rbd1.in‚0.0625" disk.nvme0c0n1.in‚ 344.70703125 disk.sda3.in‚0* proc.memory.percent‚16.412758827209473 disk.rbd0.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 483.7421875 disk.dm-0.in‚ 344.70703125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 250.296875 cpu‚26.000459013132904 disk.nvme0n1.in‚ 344.70703125 disk.sda.out‚ 250.296875 disk.sda2.in‚0 disk.sda3.out‚ 250.296875$ memory_percent‚97.04747200012207 proc.cpu.threads‚17! disk.dm-0.out‚30014.384765625 g±Ï‚(:„ „æçÒΫõ¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p×:²ÿЂ(:ô „æçÒÊò€§ disk.sda2.in‚0$ disk./.usageGB‚545.6536254882812 disk.sda3.out‚ 251.9921875 disk.dm-1.out‚ 251.9921875' disk.nvme0c0n1.out‚30029.4345703125 disk.sda.out‚ 251.9921875) disk./.usagePercent‚36.61927264040518 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 1.837467e+06 disk.dm-1.in‚0% disk.nvme0n1.out‚30029.4345703125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 345.53515625( proc.memory.availableMB‚ 483.80859375! disk.rbd1.out‚101.35205078125 disk.nvme0n1.in‚ 345.53515625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2689.06640625 cpu‚26.001103130133313 disk.sda3.in‚0 disk.sda.in‚0* proc.memory.percent‚16.412758827209473 disk.rbd3.out‚0 disk.drbd2165.in‚0# memory_percent‚97.0470666885376 network.sent‚ 1.3074519e+07 disk.rbd3.in‚0 disk.dm-0.in‚ 345.53515625" disk.dm-0.out‚30029.4345703125²^5U2Ñ- _wandbruntime‚7017  _runtime‚7017×,œùÆÒÀ _runtime‚7017.077306105 _step‚676 & lr-AdamW/pg1‚0.0003090410446614286 & lr-AdamW/pg2‚0.0003090410446614286 trainer/global_step‚8249 $ _timestamp‚1.784279824951725e+09Â9Ä0ËÓÅ _step‚676 & lr-AdamW/pg1‚0.0003090410446614286 & lr-AdamW/pg2‚0.0003090410446614286 trainer/global_step‚8249 $ _timestamp‚1.784279824951725e+09 _runtime‚7017.077306105¤.ÇžŠ2Ô- _wandbruntime‚7017  _runtime‚7017"Ô„ò'ÕìO _step‚677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76397857 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65381182 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65381182 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1262086022720000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76387880 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17769994 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141779039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276229460117504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8720012926464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17770263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1246921574002688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141769060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8817449903104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14452528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276229478729728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1262086022720000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8720012926464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8706705485067 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1253365668454400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76387880 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1253365651415040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65381180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8720354265600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784279824952761e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1262085664341504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1255628279487755 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚7017.078547905 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1253365668454400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8817530091520 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1255628637077005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65381180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76397857 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8720354265600 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1262085664341504 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚8249 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285046910020608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890쪡÷'ÖñO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285047008821248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1246921591042048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32222522 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14452531 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76387880 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1253365651415040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65381180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8720354265600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784279824952761e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1262085664341504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1255628279487755 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚7017.078547905 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1253365668454400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8817530091520 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1255628637077005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65381180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76397857 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8720354265600 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1262085664341504 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚8249 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285046910020608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76397857 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65381182 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65381182 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1262086022720000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76387880 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17769994 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141779039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276229460117504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8720012926464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17770263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1246921574002688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141769060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8817449903104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14452528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276229478729728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1262086022720000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8720012926464 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8706705485067 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1253365668454400 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8721366899712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76396932 _step‚678 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1262234276383744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8818557754880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76396932 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8721366899712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14454276 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276262888876032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32226612 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1262233917998080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141795737 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1253512909484032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8721025553408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚285081364352512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141795737 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17772336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76406923 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32226340 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1253512892444672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8707716538789 TDDeviceStatsMonitor.on_train_batch_end/reserved_byëêà tes.large_pool.peak‚ 23842521088 _runtime‚7017.692142156 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚285081364352512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141795737 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17772336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76406923 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32226340 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1253512892444672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8707716538789 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚7017.692142156 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8721366899712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76396932 _step‚678 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1262234276383744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8818557754880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76396932 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8721366899712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14454276 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276262888876032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32226612 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1262233917998080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141795737 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1253512909484032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8721025553408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76852309 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚681 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142605112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76842332 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14539778 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8868945378816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1269498411324928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65762782 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17873586 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1260727719936000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1254247107596288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1263004429568263 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14539781 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286769862034944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8868863093248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8757321971975 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8770708428288 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65762780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17873833 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76852309 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚681 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142605112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76842332 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14539778 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8868945378816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1269498411324928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65762782 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17873586 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1260727719936000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1254247107596288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1263004429568263 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14539781 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286769862034944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8868863093248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8757321971975 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8770708428288 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65762780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17873833 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14541523 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76851384 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1260874960965632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65770412 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1260874943926272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65770414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76861375 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142631789 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1254393635307520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142631789 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76851384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17875645 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8771721055232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1269646664981504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8772062401536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32417435 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8758333025697 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1269647023367168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8758673575699 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1260874960965632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277934326802432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65770412 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1260874943926272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76861375 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32417168 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1269647023367168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17875909 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8772062401536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8758333025697 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1269647023367168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8758673575699 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1260874960965632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277934326802432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65770412 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1260874943926272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76861375 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32417168 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1269647023367168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17875909 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8772062401536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14541523 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76851384 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1260874960965632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65770412 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8299 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1260874943926272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65770414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76861375 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142631789 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1254393635307520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142631789 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76851384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17875645 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ªùd}Ã2ì- _wandbruntime‚7058  _runtime‚705887¥ í… 0loss/FRt1/assignment_loss‚1.6512085256027162 3loss/SRqqt2/assignment_loss‚0.48598447821954677 /loss/FRt1/detection_loss‚0.5347574639686854 /loss/FRt2/detection_loss‚1.2729477354193348 2loss/SRqqt1/assignment_loss‚0.4954600059781755 0loss/FBt2/detection_loss‚0.46681950067079836 _runtime‚7058.501496167 _step‚683 4loss/FBt2/assignment_loss‚0.00015971236367172718 1loss/SRqqt1/detection_loss‚0.9277210883081022 /loss/FBt1/detection_loss‚0.4312523000616469 epoch‚20 trainer/global_step‚8299 0loss/FRt2/assignment_loss‚0.8401781505772046 2loss/FBt1/assignment_loss‚0.010819886656451935 0loss/SRqqt2/detection_loss‚0.560105562792306 $loss/total_loss‚7.67741441061864 $ _timestamp‚1.784279866376158e+09ŽÓ8îŠ 0loss/SRqqt2/detection_loss‚0.560105562792306 $loss/total_loss‚7.67741441061864 $ _timestamp‚1.784279866376158e+09 0loss/FRt1/assignment_loss‚1.6512085256027162 3loss/SRqqt2/assignment_loss‚0.48598447821954677 /loss/FRt1/detection_loss‚0.5347574639686854 /loss/FRt2/detection_loss‚1.2729477354193348 2loss/SRqqt1/assignment_loss‚0.4954600059781755 0loss/FBt2/detection_loss‚0.46681950067079836 _runtime‚7058.501496167 _step‚683 4loss/FBt2/assignment_loss‚0.00015971236367172718 1loss/SRqqt1/detection_loss‚0.9277210883081022 /loss/FBt1/detection_loss‚0.4312523000616469 epoch‚20 trainer/global_step‚8299 0loss/FRt2/assignment_loss‚0.8401781505772046 2loss/FBt1/assignment_loss‚0.010819886656451935«ã¯“ï‚(:ˆ ÀæçÒÚ°£¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚265.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Lã«ùð‚(:î ÀæçÒôƒ›§) disk./.usagePercent‚36.61926931230723 disk.dm-0.in‚ 347.91796875 disk.drbd2165.in‚0# memory_percent‚97.0496416091919 proc.cpu.threads‚17 disk.rbd1.in‚0.0625! disk.rbd1.out‚101.47705078125 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 network.sent‚ 1.3163477e+07 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 347.91796875 disk.dm-1.in‚0 disk.dm-1.out‚ 258.85546875 network.recv‚ 1.851592e+06" disk.nvme0n1.out‚ 30075.2890625 disk.sda.in‚0 disk.sda.out‚ 258.85546875 disk.sda3.in‚0 disk.dm-0.out‚ 30075.2890625 disk.sda1.out‚0( proc.memory.availableMB‚ 483.38671875* proc.memory.percent‚16.412758827209473$ disk./.usageGB‚545.6535758972168 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 30075.2890625 disk.sda2.in‚0 disk.sda3.out‚ 258.85546875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 347.91796875 disk.rbd0.out‚0 cpu‚26.004397223394637# proc.memory.rssMB‚ 2689.06640625ÿë–Ž“ñ‚(:ˆ ÏæçÒæþº¢ gpu.0.gpu‚98.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚248.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì5øþò‚(:ó ÏæçÒËÄó¥ disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 261.1328125$ disk./.usageGB‚545.6494216918945 disk.sda1.out‚0 disk.sda.out‚ 261.1328125 disk.rbd3.in‚0 disk.nvme0n1.in‚ 348.74609375& disk.nvme0n1.out‚30095.47705078125 disk.drbd2165.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚30095.47705078125 disk.rbd0.in‚0! disk.rbd1.out‚101.53955078125# proc.memory.rssMB‚ 2689.06640625 cpu‚26.005275175224625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.in‚0.0625 network.recv‚ 1.856359e+06( disk.nvme0c0n1.out‚30095.47705078125 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚97.04971313476562* proc.memory.percent‚16.412758827209473 disk.sda1.in‚0 disk.dm-1.out‚ 261.1328125 disk.dm-0.in‚ 348.74609375# proc.memory.availableMB‚483.375 disk.sda.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 348.74609375 network.sent‚ 1.3215625e+07) disk./.usagePercent‚36.618990520102453@ÈF“ó‚(:ˆ ÞæçÒ¤ü¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÌÅþô‚(:ó ÞæçÒœ×÷¦* proc.memory.percent‚16.412758827209473 cpu‚26.00581626522386" disk.nvme0c0n1.in‚ 348.74609375( disk.nvme0c0n1.out‚30103.77783203125 disk.sda2.out‚0 disk.sda3.out‚ 262.4765625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 348.74609375# disk.dm-0.out‚30103.77783203125* disk./.usagePercent‚36.618770353623006 disk.rbd0.out‚0 disk.sda.out‚ 262.4765625 disk.dm-1.out‚ 262.4765625 disk.rbd1.in‚0.0625 disk.sda1.in‚0$ disk./.usageGB‚545.6461410522461 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 348.74609375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0# proc.memory.rssMB‚ 2689.06640625 network.recv‚ 1.859509e+06 disk.sda.in‚0 disk.sda2.in‚0! disk.rbd1.out‚101.53955078125& disk.nvme0n1.out‚30103.77783203125$ memory_percent‚97.04971313476562 network.sent‚ 1.3220773e+07 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.availableMB‚483.375H¸Ãî2õ- _wandbruntime‚7098  _runtime‚7098ÒIU¬ÆöÀ & lr-AdamW/pg1‚0.0003054274876141448 & lr-AdamW/pg2‚0.0003054274876141448 trainer/global_step‚8349 $ _timestamp‚1.784279906237063e+09 _runtime‚7098.362622816 _step‚684˜ÙÇË÷Å & lr-AdamW/pg1‚0.0003054274876141448 & lr-AdamW/pg2‚0.0003054274876141448 trainer/global_step‚8349 $ _timestamp‚1.784279906237063e+09 _runtime‚7098.362622816 _step‚684¬ÊL2ø- _wandbruntime‚7098  _runtime‚7098ó†’ùùîO aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1268089771417600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8821403930112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8821745269248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1261572641189888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1276911158308352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77296784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14627031 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1268089754378240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8821403930112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288492695689216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1276911516686848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288492594791424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77296784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17978003 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8821745269248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66144380 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1276911158308352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 i„ù'úóO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1268089771417600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8821403930112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8821745269248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1261572641189888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1276911158308352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77296784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14627031 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1268089754378240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8821403930112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288492695689216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1276911516686848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288492594791424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77296784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17978003 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8821745269248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66144380 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1276911158308352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376­¾;Ú2û- _wandbruntime‚7098  _runtime‚7098AËùý&ü÷M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261719168901120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8808949512605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14628773 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32608856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66152012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32608589 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143467841 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143467841 % _timestamp‚1.7842799068500385e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1270528101374365 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77305836 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1268237012447232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288527150992384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66152014 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1268237012447232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1268236995407872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66152012 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17980080 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1277059770350592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288527054296064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1277059411964928 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1268236995407872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14628776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143457848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8921289567232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1261719151861760 _step‚686 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77315827 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8822416557056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143457848 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8921367651328 trainer/global_step‚8349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77315827 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8822416557056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605783341056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279605764728832çOÌWº ýüM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32608856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66152012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32608589 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143467841 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143467841 % _timestamp‚1.7842799068500385e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1270528101374365 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77305836 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1268237012447232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288527150992384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66152014 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1268237012447232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1268236995407872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66152012 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17980080 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1277059770350592 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288527054296064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1277059411964928 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1268236995407872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14628776 ;4DeviceStatsMo³_±ÑHnitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143457848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8921289567232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1261719151861760 _step‚686 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77315827 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8822416557056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143457848 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8921367651328 trainer/global_step‚8349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77315827 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8822416557056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605783341056 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279605764728832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261719168901120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8808949512605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14628773 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169®"¢½2þ- _wandbruntime‚7098  _runtime‚7098³?Ïÿ 2loss/SRqqt1/assignment_loss‚0.4890398495668189 .loss/FBt1/detection_loss‚0.482644783062357 epoch‚20 0loss/FRt2/assignment_loss‚0.9375356115973913 3loss/SRqqt2/assignment_loss‚0.44231516030915735 1loss/FBt1/assignment_loss‚0.01133288679696089 /loss/FRt2/detection_loss‚1.1728603705574643 /loss/FBt2/detection_loss‚0.8639603098563384 %loss/total_loss‚8.127344437989537 trainer/global_step‚8349 $ _timestamp‚1.784279906852741e+09 3loss/FBt2/assignment_loss‚0.0014982885052585715 1loss/SRqqt1/detection_loss‚0.9075583878697621 1loss/SRqqt2/detection_loss‚0.6408445180388175 _runtime‚ 7098.97806525 _step‚687 /loss/FRt1/detection_loss‚0.5320486582836861 /loss/FRt1/assignment_loss‚1.645705613545525^ -õ €† /loss/FRt1/assignment_loss‚1.645705613545525 2loss/SRqqt1/assignment_loss‚0.4890398495668189 .loss/FBt1/detection_loss‚0.482644783062357 epoch‚20 0loss/FRt2/assignment_loss‚0.9375356115973913 3loss/SRqqt2/assignment_loss‚0.44231516030915735 1loss/FBt1/assignment_loss‚0.01133288679696089 /loss/FRt2/detection_loss‚1.1728603705574643 /loss/FBt2/detection_loss‚0.8639603098563384 %loss/total_loss‚8.127344437989537 trainer/global_step‚8349 $ _timestamp‚1.784279906852741e+09 3loss/FBt2/assignment_loss‚0.0014982885052585715 1loss/SRqqt1/detection_loss‚0.9075583878697621 1loss/SRqqt2/detection_loss‚0.6408445180388175 _runtime‚ 7098.97806525 _step‚687 /loss/FRt1/detection_loss‚0.5320486582836861¯Q=Ó1‘‚(:† íæçÒûµõ¤ gpu.0.gpu‚90.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚202.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c6¯‚‚(:ø íæçÒÞ„ü¦* disk./.usagePercent‚36.618871988614096 disk.nvme0n1.in‚ 349.4609375 disk.rbd0.out‚0 disk.sda.out‚ 264.73046875 network.sent‚ 1.3223695e+07$ disk./.usageGB‚545.6476554870605 disk.dm-1.out‚ 264.73046875 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚30116.01416015625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2689.06640625 disk.sda3.in‚0 disk.sda3.out‚ 264.73046875 disk.rbd1.in‚0.0625! disk.rbd1.out‚101.53955078125* proc.memory.percent‚16.412758827209473 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.in‚0$ memory_percent‚97.05088138580322 disk.sda2.in‚0# disk.dm-0.out‚30116.01416015625 disk.drbd2165.in‚0 disk.rbd0.in‚0 network.recv‚ 1.862861e+06! disk.nvme0c0n1.in‚ 349.4609375 disk.dm-1.in‚0 disk.dm-0.in‚ 349.4609375( disk.nvme0c0n1.out‚30116.01416015625( proc.memory.availableMB‚ 483.18359375 cpu‚26.00653751180144 disk.sda1.out‚0[¼¢ “ƒ‚(:ˆ üæçÒݨҤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û¡á©„‚(:ø üæçÒЖǦ( disk.nvme0c0n1.out‚30129.48583984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 network.recv‚ 1.867111e+06$ memory_percent‚97.05054759979248 network.sent‚ 1.3286636e+07$ disk./.usageGB‚545.6477432250977 disk.sda1.in‚0! disk.rbd1.out‚101.60205078125 disk.dm-0.in‚ 350.2890625( proc.memory.availableMB‚ 483.23828125 disk.sda1.out‚0 disk.sda.out‚ 265.92578125# proc.memory.rssMB‚ 2689.06640625 proc.cpu.threads‚17 disk.sda.in‚0! disk.nvme0c0n1.in‚ 350.2890625# disk.dm-0.out‚30129.48583984375* proc.memory.percent‚16.412758827209473 cpu‚26.007119698948134& disk.nvme0n1.out‚30129.48583984375 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 265.92578125 disk.rbd1.in‚0.0625 disk.rbd3.in‚0) disk./.usagePercent‚36.61887787678738 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 265.92578125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 350.2890625 disk.rbd0.out‚0ÛÉ82…- _wandbruntime‚7138  _runtime‚7138Q½¨›È†Â _step‚688 ' lr-AdamW/pg1‚0.00030361399438628457 ' lr-AdamW/pg2‚0.00030361399438628457 trainer/global_step‚8399 % _timestamp‚1.7842799467947478e+09 _runtime‚ 7138.92034709Ÿ ÅÍ‡Ç _step‚688 ' lr-AdamW/pg1‚0.00030361399438628457 ' lr-AdamW/pg2‚0.00030361399438628457 trainer/global_step‚8399 % _timestamp‚1.7842799467947478e+09 _runtime‚ 7138.92034709°Y=tp2ˆ- _wandbruntime‚7138  _runtime‚7138Ñêÿô'‰îO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290215508050432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66525982 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8872440771072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1275451822899200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8872099431936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66525982 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1277756729729279 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281243792508928 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8858895495681 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32795800 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8971734153728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1275451822899200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1284323905291776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77761213 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66525980 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7842799467957535e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1275451805859840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚689 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1284324263670272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77761213 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144277216 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1284324263670272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8872099431936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14714281 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281243773896704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8971818536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1284323905291776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66525980 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8872440771072 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144287195 JÊ´ù'ŠóO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1275451822899200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1284323905291776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77761213 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66525980 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7842799467957535e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1275451805859840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚689 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1284324263670272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77761213 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144277216 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1284324263670272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8872099431936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14714281 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281243773896704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8971818536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1284323905291776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66525980 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8872440771072 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144287195 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8858895495681 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32795800 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8971734153728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088±ýA(2‹- _wandbruntime‚7139  _runtime‚7139xnò«ÁŒ÷M ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144303893 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStats(oL}<$Monitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1284472517334016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144293900 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14716023 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚7139.534826838 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873112058880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281277221267456 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66533614 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32799897 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 trainer/global_step‚8399 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8873453405184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873112058880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1275599063928832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144293900 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1269044685455360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8873453405184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1269044702494720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77760288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290250062326784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77770279 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8859565999513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8972750392320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.784279947409025e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144303893 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18083614 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77770279 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32799637 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1277904251454873 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66533612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1284472517334016 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1275599046889472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66533614 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14716026 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8972841059328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8859906549515 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚7139.534826838 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873112058880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281277221267456 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66533614 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32799897 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 trainer/global_step‚8399 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8873453405184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873112058880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1275599063928832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144293900 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1269044685455360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8873453405184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1269044702494720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77760288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290250062326784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77770279 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8859565999513 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8972750392320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.784279947409025e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144303893 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18083614 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77770279 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32799637 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1277904251454873 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66533612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1284472517334016 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1275599046889472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66533614 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14716026 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8972841059328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8859906549515 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7471465468406677 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9406041502952576 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.505125880241394 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24003371596336365 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988139867782593 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211727380752563 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5672674775123596 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23685649037361145 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7422440648078918 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 particle/accuracy_4_of_5‚1 _step‚692 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.801058292388916 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_2‚0.8179933428764343 particle/accuracy_5_of_5‚1 epoch‚20 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8692822456359863 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030187401920557022 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.31277161836624146 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7428401708602905 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5061842799186707 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028870727866888046 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6996856927871704 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6600926518440247 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5850844383239746 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5238146185874939 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7889046669006348 trainer/global_step‚8399 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957537651062012 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.660071849822998 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7870533466339111 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7712284922599792 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6573201417922974 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2630256116390228 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6940497159957886 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27549177408218384 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8532803058624268 &particle/f_score‚0.643222987651825 % _timestamp‚1.7842799654578786e+09 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17195861041545868 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.690003514289856 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.31277161836624146 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9100178480148315 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8252079486846924 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7981690764427185 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_4‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8207294940948486 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16750000417232513 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10826350748538971 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8101851940155029 /particle/accuracy_4_of_4‚0.9514333009719849 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6744339466094971 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03045240230858326 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955635070800781 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99690181016922 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05654105171561241 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3231070637702942 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028870727866888046 +jet/accuracy_3_of_4‚0.08379949629306793 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5209392309188843 )loss/val_total_loss‚8.011671680964223 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6536757349967957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964742064476013 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0¾ãSD¸027204242069274187 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9956644773483276 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05785619467496872 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5479167103767395 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8532803058624268 *jet/accuracy_1_of_4‚0.9748600721359253 (particle/accuracy‚0.8488070964813232 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 *jet/accuracy_2_of_4‚0.8090664744377136 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7156023979187012 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6370429396629333 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6651609539985657 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2766153812408447 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7174332737922668 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 5validation_average_jet_accuracy‚0.404923677444458 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18768198788166046 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6670947074890137 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.724784791469574 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26455920934677124 *jet/accuracy_1_of_1‚0.3670779764652252 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7438951134681702 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.6029887795448303 *particle/specificity‚0.907342255115509 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.665364682674408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36539965867996216 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6775922179222107 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07611972093582153 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.990498423576355 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9301355481147766 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9921571612358093 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893808960914612 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7159500122070312 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7926186919212341 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9979296326637268 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030736753717064857 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7652112245559692 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.754510223865509 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17268189787864685 +jet/accuracy_2_of_2‚0.16669805347919464 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27352872490882874 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9869707822799683 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8007748126983643 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24378710985183716 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09419604390859604 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04602733999490738 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2983665466308594 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _runtime‚7157.584146705 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5389959216117859 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7125927805900574 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3225104808807373 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6980487704277039 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6670947074890137 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7560918927192688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33845630288124084 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985955536365509 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5965533256530762 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8655712604522705 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04920743778347969 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3971375524997711 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8207294940948486 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7316890358924866 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30338311195373535 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 •T>r=—ìz 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211727380752563 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5672674775123596 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23685649037361145 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7422440648078918 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 particle/accuracy_4_of_5‚1 _step‚692 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.801058292388916 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_2‚0.8179933428764343 particle/accuracy_5_of_5‚1 epoch‚20 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8692822456359863 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030187401920557022 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.31277161836624146 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7428401708602905 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5061842799186707 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028870727866888046 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6996856927871704 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6600926518440247 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5850844383239746 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5238146185874939 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7889046669006348 trainer/global_step‚8399 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957537651062012 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.660071849822998 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7870533466339111 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7712284922599792 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6573201417922974 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2630256116390228 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6940497159957886 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27549177408218384 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8532803058624268 &particle/f_score‚0.643222987651825 % _timestamp‚1.7842799654578786e+09 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17195861041545868 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.690003514289856 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.31277161836624146 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9100178480148315 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8252079486846924 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7981690764427185 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_4‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8207294940948486 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16750000417232513 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10826350748538971 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8101851940155029 /particle/accuracy_4_of_4‚0.9514333009719849 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6744339466094971 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03045240230858326 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955635070800781 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99690181016922 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05654105171561241 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3231070637702942 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028870727866888046 +jet/accuracy_3_of_4‚0.08379949629306793 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5209392309188843 )loss/val_total_loss‚8.011671680964223 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6536757349967957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964742064476013 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9956644773483276 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05785619467496872 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5479167103767395 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8532803058624268 *jet/accuracy_1_of_4‚0.9748600721359253 (particle/accuracy‚0.8488070964813232 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 *jet/accuracy_2_of_4‚0.8090664744377136 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7156023979187012 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6370429396629333 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6651609539985657 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2766153812408447 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7174332737922668 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 5validation_average_jet_accuracy‚0.404923677444458 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18768198788166046 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6670947074890137 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.724784791469574 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26455920934677124 *jet/accuracy_1_of_1‚0.3670779764652252 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7438951134681702 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.6029887795448303 *particle/specificity‚0.907342255115509 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.665364682674408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36539965867996216 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6775922179222107 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07611972093582153 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.990498423576355 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9301355481147766 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9921571612358093 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893808960914612 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7159500122070312 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7926186919212341 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9979296326637268 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030736753717064857 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7652112245559692 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.754510223865509 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17268189787864685 +jet/accuracy_2_of_2‚0.16669805347919464 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27352872490882874 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9869707822799683 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8007748126983643 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24378710985183716 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09419604390859604 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04602733999490738 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2983665466308594 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _runtime‚7157.584146705 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5389959216117859 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7125927805900574 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3225104808807373 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6980487704277039 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6670947074890137 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7560918927192688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33845630288124084 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985955536365509 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5965533256530762 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8655712604522705 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04920743778347969 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3971375524997711 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8207294940948486 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7316890358924866 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30338311195373535 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988561272621155 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6605353355407715 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3230040967464447 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6646069884300232 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15690003335475922 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32110708951950073 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030987534672021866 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787617921829224 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3716265559196472 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39931005239486694 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2622910439968109 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7568248510360718 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31347617506980896 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.457102507352829 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23562131822109222 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24641475081443787 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2577414810657501 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7328524589538574 particle/accuracy_1_of_3‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3117733895778656 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8101851940155029 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_3_of_3‚1 +particle/sensitivity‚0.6347336769104004 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7471465468406677 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9406041502952576 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.505125880241394 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24003371596336365 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988139867782593´š+Ež˜‚(:“ ©ççÒò gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yâL¼ù™‚(:î ©ççÒöÅħ network.recv‚ 1.879531e+06 disk.nvme0n1.in‚ 351.83203125% disk.nvme0n1.out‚30163.7685546875 disk.sda.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚36.61862161324561 disk.sda1.out‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 434.546875$ disk./.usageGB‚545.6439247131348" disk.nvme0c0n1.in‚ 351.83203125 disk.sda2.in‚0 disk.dm-1.out‚ 269.265625 disk.rbd3.out‚0 disk.dm-0.in‚ 351.83203125$ memory_percent‚97.34773635864258* proc.memory.percent‚16.701602935791016 disk.rbd0.out‚0 cpu‚26.017773913376804 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 269.265625 disk.sda2.out‚0" disk.dm-0.out‚30163.7685546875! proc.memory.rssMB‚ 2736.390625 network.sent‚ 1.3373242e+07' disk.nvme0c0n1.out‚30163.7255859375 disk.sda3.out‚ 269.265625 disk.dm-1.in‚0 disk.sda1.in‚0! disk.rbd1.out‚101.66455078125íáe½“š‚(:ˆ ¸ççÒàÕ¤ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚247.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý°9’ü›‚(:ñ ¸ççÒïî¦ disk.drbd2165.out‚0 network.recv‚ 1.889392e+06 disk.sda3.out‚ 270.93359375' proc.memory.availableMB‚ 435.5546875 network.sent‚ 1.3420473e+07 disk.dm-0.in‚ 352.546875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0! disk.rbd1.out‚101.72705078125! proc.memory.rssMB‚ 2736.390625 disk.nvme0n1.in‚ 352.546875& disk.nvme0n1.out‚30176.71826171875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 352.546875 disk.sda3.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.34158515930176 cpu‚26.018468404633083# disk.dm-0.out‚30176.71826171875( disk.nvme0c0n1.out‚30176.71826171875 disk.dm-1.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚36.6186095808915 disk.dm-1.out‚ 270.93359375 disk.rbd3.out‚0$ disk./.usageGB‚545.6437454223633 disk.sda.in‚0 disk.sda.out‚ 270.93359375 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.701602935791016šÚÁo2œ- _wandbruntime‚7197  _runtime‚7197’Ö‘ÆÀ % _timestamp‚1.7842800053696592e+09 _runtime‚ 7197.49645302 _step‚693 & lr-AdamW/pg1‚0.0003017961798969779 & lr-AdamW/pg2‚0.0003017961798969779 trainer/global_step‚8449F3§=˞Š% _timestamp‚1.7842800053696592e+09 _runtime‚ 7197.49645302 _step‚693 & lr-AdamW/pg1‚0.0003017961798969779 & lr-AdamW/pg2‚0.0003017961798969779 trainer/global_step‚8449µî­X%2Ÿ- _wandbruntime‚7197  _runtime‚7197Íá²n îO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67028246 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33020751 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291972101141504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8926891157928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78285987 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8927231707818 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14804048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67028248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282931123922944 % _timestamp‚1.7842800053703654e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1284743423991808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145304256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1287059620338346 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18216964 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291972206233600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _runtime‚7197.496560233 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1284743441031168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8940528139264 TGDeviceStatsMonitor.on_train_batch_start/inact£÷6ö† ive_split.small_pool.freed‚18216703 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9040977218560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67028246 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1287059262749096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14804051 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1293683952131072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78285987 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8940528139264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1278132388630528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269Sݤù'¡óO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1284743441031168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8940528139264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18216703 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9040977218560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67028246 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1287059262749096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14804051 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1293683952131072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78285987 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8940528139264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1278132388630528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67028246 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33020751 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291972101141504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8926891157928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78285987 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8927231707818 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14804048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67028248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282931123922944 % _timestamp‚1.7842800053703654e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1284743423991808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145304256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1287059620338346 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18216964 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291972206233600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _runtime‚7197.496560233¶V…Å2¢- _wandbruntime‚7198  _runtime‚7198U0ý&£÷M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1284890682060800 _step‚695 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145330933 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8941540766208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1284890682060800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145330933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1293832205787648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8941882112512 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33024494 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18218957 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1287206784474690 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78285062 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1293832564173312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292006566852096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78295053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14805793 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292006665645568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚7198.108094052 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8941882112512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1293832564173312 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1284890665021440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9042094351872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9042014170624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1278278899302400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282964571293696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362HRÐ?/#¤üM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33024494 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18218957 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1287206784474690 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78285062 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1293832564173312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292006566852096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78295053 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14805793 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292006665645568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚7198.108094052 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8941882112512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1293832564173312 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1284890665021440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9042094351872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9042014170624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1278278899302400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282964571293696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1284890682060800 _step‚695 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145330933 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8941540766208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1284890682060800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145330933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1293832205787648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8941882112512 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1301097057492992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146150287 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚698 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9092219357184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67409846 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1301096699114496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚293694896784384 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚7238.026547901 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8991223641088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18320436 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18320715 trainer/global_step‚8499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78740439 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293694781206528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1301096699114496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284602580461568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78740439 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14891298 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1294435412829604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8991564980224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1301097057492992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146150287 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚698 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9092219357184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67409846 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1301096699114496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚293694896784384 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚7238.026547901 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8991223641088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18320436 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18320715 trainer/global_step‚8499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78740439 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293694781206528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1301096699114496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284602580461568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78740439 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14891298 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1294435412829604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8991564980224 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚8499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146156992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67417480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8992577614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1285604432896000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842800465127325e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78749505 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33215435 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1301245311156736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8992236268032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18322665 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293729345628672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8978859248560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚7238.638488109 _step‚699 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14893043 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146166985 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14893046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284636009220096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1292252733542400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1294583292144560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1301244952771072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146156992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78739514 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146166985 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1292252716503040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072>>˜'¶ýM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293729227960832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284635990607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1294582934555198 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚8499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146156992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67417480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8992577614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1285604432896000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842800465127325e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78749505 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33215435 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1301245311156736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8992236268032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18322665 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293729345628672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8978859248560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚7238.638488109 _step‚699 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14893043 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146166985 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14893046 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284636009220096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1292252733542400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1294583292144560 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1301244952771072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146156992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78739514 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146166985 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1292252716503040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1301245311156736 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1292252733542400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1285604415856640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8992577614336»càv‚2·- _wandbruntime‚7238  _runtime‚7238Ý™ô˜¸€ 0loss/FRt2/assignment_loss‚0.9371309753006963 3loss/SRqqt1/assignment_loss‚0.39340286586928885 /loss/FBt2/detection_loss‚0.5396605775017163 _step‚700 0loss/FRt1/assignment_loss‚1.6417114764350726 2loss/FBt1/assignment_loss‚0.016594723139646626 /loss/FRt2/detection_loss‚1.3005114392709307 /loss/FBt1/detection_loss‚0.4225853638084972 _runtime‚ 7238.64175165 0loss/FBt2/assignment_loss‚0.1036246822973363 /loss/FRt1/detection_loss‚0.4861415219494199 /loss/SRqqt1/detection_loss‚1.10146023390418 epoch‚21 % _timestamp‚1.7842800465164294e+09 2loss/SRqqt2/assignment_loss‚0.5029844992919281 1loss/SRqqt2/detection_loss‚0.5810148871744589 %loss/total_loss‚8.026823245943172 trainer/global_step‚8499Ãt$ ¹… 0loss/FRt1/assignment_loss‚1.6417114764350726 2loss/FBt1/assignment_loss‚0.016594723139646626 /loss/FRt2/detection_loss‚1.3005114392709307 /loss/FBt1/detection_loss‚0.4225853638084972 _runtime‚ 7238.64175165 0loss/FBt2/assignment_loss‚0.1036246822973363 /loss/FRt1/detection_loss‚0.4861415219494199 /loss/SRqqt1/detection_loss‚1.10146023390418 epoch‚21 % _timestamp‚1.7842800465164294e+09 2loss/SRqqt2/assignment_loss‚0.5029844992919281 1loss/SRqqt2/detection_loss‚0.5810148871744589 %loss/total_loss‚8.026823245943172 trainer/global_step‚8499 0loss/FRt2/assignment_loss‚0.9371309753006963 3loss/SRqqt1/assignment_loss‚0.39340286586928885 /loss/FBt2/detection_loss‚0.5396605775017163 _step‚700¼Û…3’º‚(:‡ ôççÒ÷‡¬¥ gpu.0.gpu‚90.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚254.805' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 àÙ}ù»‚(:î ôççÒ®½ó§ disk.nvme0n1.in‚ 354.91796875 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚26.02101759752897 proc.cpu.threads‚17 network.sent‚ 1.3498915e+07) disk./.usagePercent‚36.61877060963054 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda3.out‚ 272.01171875 disk.rbd3.in‚0 disk.sda1.in‚0$ memory_percent‚97.34456539154053* proc.memory.percent‚16.701602935791016 disk.dm-1.out‚ 272.01171875$ disk./.usageGB‚545.6461448669434% disk.nvme0c0n1.out‚30207.48046875 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd0.out‚0( proc.memory.availableMB‚ 435.06640625! disk.rbd1.out‚101.85205078125 disk.rbd3.out‚0 disk.dm-0.out‚30207.48046875! proc.memory.rssMB‚ 2736.390625 disk.dm-0.in‚ 354.91796875# disk.nvme0n1.out‚30207.48046875 disk.drbd2165.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 354.91796875 disk.sda.out‚ 272.01171875 disk.sda3.in‚0 disk.dm-1.in‚0 network.recv‚ 1.90425e+06è5Ë“¼‚(:ˆ ƒèç҈·¤ gpu.0.gpu‚62.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øXÜ?½‚(:÷ ƒèçÒ¼Ýܦ network.recv‚ 1.916654e+06 disk.dm-0.in‚ 355.6328125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚30213.87744140625 disk.sda.out‚ 272.25390625 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 435.05859375 cpu‚26.021736007973796 disk.sda.in‚0 disk.sda2.out‚0# disk.dm-0.out‚30213.87744140625 disk.nvme0n1.in‚ 355.6328125$ memory_percent‚97.34461307525635 disk.sda3.out‚ 272.25390625! disk.rbd1.out‚101.85205078125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.3565182e+07* disk./.usagePercent‚36.618778289856564$ disk./.usageGB‚545.6462593078613 disk.rbd3.in‚0& disk.nvme0n1.out‚30213.87744140625* proc.memory.percent‚16.701602935791016 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0! proc.memory.rssMB‚ 2736.390625 disk.dm-1.in‚0 disk.dm-1.out‚ 272.25390625 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 355.63281258Ø×’¾‚(:‡ ’èçÒ‘î¼¢ gpu.0.gpu‚93.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚254.525' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.905 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&SsÒò¿‚(:ç ’èçÒÉ€ó¥ disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 cpu‚26.02224397444699 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0! disk.rbd1.out‚101.91455078125 disk.nvme0n1.in‚ 356.4609375! disk.nvme0c0n1.in‚ 356.4609375$ memory_percent‚97.34659194946289 network.sent‚ 1.3572367e+07 disk.sda3.out‚ 272.65625 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 272.65625" disk.dm-0.out‚30222.9736328125 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 434.734375$ disk./.usageGB‚545.6461944580078 disk.rbd1.in‚0.0625% disk.nvme0n1.out‚30222.9736328125' disk.nvme0c0n1.out‚30222.9736328125! proc.memory.rssMB‚ 2736.390625* proc.memory.percent‚16.701602935791016 network.recv‚ 1.91995e+06* disk./.usagePercent‚36.618773937728484 disk.dm-0.in‚ 356.4609375 disk.sda.out‚ 272.65625 proc.cpu.threads‚17ÖùT“2À- _wandbruntime‚7278  _runtime‚7278–SÖÈÁ ' lr-AdamW/pg1‚0.00029814805442338396 ' lr-AdamW/pg2‚0.00029814805442338396 trainer/global_step‚8549 $ _timestamp‚1.784280086304743e+09 _runtime‚7278.430422923 _step‚701=¤´ÍÂÇ ' lr-AdamW/pg2‚0.00029814805442338396 trainer/global_step‚8549 $ _timestamp‚1.784280086304743e+09 _runtime‚7278.430422923 _step‚701 ' lr-AdamW/pg1‚0.00029814805442338396½¢ŽÍÙ2Ã- _wandbruntime‚7278  _runtime‚7278_Uaãô'ÄîO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67791446 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9028464681634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842800863059413e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33401683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9143228224000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1308509804476416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9041919142912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1299467543994368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67791446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14978548 _runtime‚7278.431712027 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1292783455817728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146976360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9042260482048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79184914 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9042260482048 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1299467543994368 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18422869 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9028124131744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146986339 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33401417 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1308509804476416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14978551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79184914 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9041919142912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79194891 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18423132 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67791448 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1308509446097920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295417227999744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚8549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1308509446097920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1301811920499362 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1292783438778368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚702 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1299467526955008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67791448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286274018387968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1301811562910112åÙÝ5ZÅóO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9041919142912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1299467543994368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67791446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14978548 _runtime‚7278.431712027 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1292783455817728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚ͦ½Ÿ$2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146976360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9042260482048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79184914 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9042260482048 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1299467543994368 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18422869 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9028124131744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146986339 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33401417 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1308509804476416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14978551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79184914 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9041919142912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79194891 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18423132 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67791448 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1308509446097920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295417227999744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚8549 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1308509446097920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1301811920499362 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1292783438778368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚702 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1299467526955008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67791448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286274018387968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1301811562910112 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67791446 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9028464681634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842800863059413e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33401683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9143228224000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1308509804476416¾î2Æ- _wandbruntime‚7279  _runtime‚7279WOtþ&ÇøM QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1299614785024000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286307447146496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147003037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1292929949450240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚8549 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147003037 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1308658058140160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79193966 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1308658058140160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚7279.045465317 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33405216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1301959442225068 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67799080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295451783197696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18425194 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286307428534272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14980296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9042931769856 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1299614767984640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1308657699754496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚703 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79193966 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14980293 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1299614785024000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286307447146496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147003037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1292929949450240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚8549 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147003037 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1308658058140160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79193966 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1308658058140160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚7279.045465317 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33405216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1301959442225068 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67799080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295451783197696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18425194 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286307428534272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888¿ °‹¯2É- _wandbruntime‚7279  _runtime‚7279»Òg Ê… 0loss/FBt1/detection_loss‚0.39580580616700367 epoch‚21 2loss/FBt1/assignment_loss‚0.021838192641093265 _step‚704 1loss/SRqqt1/assignment_loss‚0.513345727152294 2loss/SRqqt2/assignment_loss‚0.5113204410762343 4loss/FBt2/assignment_loss‚0.00036907177775447053 /loss/FRt2/detection_loss‚1.3808351150714149 _runtime‚7279.047397785 /loss/FRt1/detection_loss‚0.5584299858690598 /loss/FBt2/detection_loss‚0.7448253797774669 %loss/total_loss‚8.233940856515256 trainer/global_step‚8549 % _timestamp‚1.7842800869219882e+09 0loss/FRt1/assignment_loss‚1.6521762677850018 0loss/FRt2/assignment_loss‚0.8235748075022555 1loss/SRqqt1/detection_loss‚0.9977028980639165 0loss/SRqqt2/detection_loss‚0.633717163631761È ×?ËŠ 0loss/FRt1/assignment_loss‚1.6521762677850018 0loss/FRt2/assignment_loss‚0.8235748075022555 1loss/SRqqt1/detection_loss‚0.9977028980639165 0loss/SRqqt2/detection_loss‚0.633717163631761 0loss/FBt1/detection_loss‚0.39580580616700367 epoch‚21 2loss/FBt1/assignment_loss‚0.021838192641093265 _step‚704 1loss/SRqqt1/assignment_loss‚0.513345727152294 2loss/SRqqt2/assignment_loss‚0.5113204410762343 4loss/FBt2/assignment_loss‚0.00036907177775447053 /loss/FRt2/detection_loss‚1.3808351150714149 _runtime‚7279.047397785 /loss/FRt1/detection_loss‚0.5584299858690598 /loss/FBt2/detection_loss‚0.7448253797774669 %loss/total_loss‚8.233940856515256 trainer/global_step‚8549 % _timestamp‚1.7842800869219882e+09ÀÚÒ¶¿žÌ‚(:“ ¡èçÒÓ“È¥ gpu.0.gpu‚76.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[j½¯Í‚(:÷ ¡èçÒø¦" disk.nvme0c0n1.in‚ 357.17578125 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 272.8984375 disk.rbd3.out‚0 disk.sda.out‚ 272.8984375# disk.dm-0.out‚30229.97607421875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2736.390625 disk.nvme0n1.in‚ 357.17578125 disk.dm-0.in‚ 357.17578125( proc.memory.availableMB‚ 434.74609375 proc.cpu.threads‚17$ disk./.usageGB‚545.6466865539551( disk.nvme0c0n1.out‚30229.97607421875 disk.sda2.out‚0& disk.nvme0n1.out‚30229.97607421875 disk.sda.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚101.91455078125* proc.memory.percent‚16.701602935791016 network.recv‚ 1.923414e+06 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 272.8984375$ memory_percent‚97.34652042388916 cpu‚26.022906411684517 network.sent‚ 1.3575331e+07* disk./.usagePercent‚36.618806962700404¸†GÐû΂(:’ °èçÒ‹ÉŤ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.95' gpu.0.enforcedPowerLimitWatts‚500.0) gžQÚ¢pu.0.powerPercent‚53.589999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cl©íÏ‚(:â °èçÒúÓë¦ disk.sda1.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚102.03955078125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 357.17578125 disk.nvme0n1.in‚ 357.17578125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚273.3125$ proc.memory.availableMB‚434.5625! proc.memory.rssMB‚ 2736.390625 network.sent‚ 1.3640331e+07) disk./.usagePercent‚36.61879902646684$ disk./.usageGB‚545.6465682983398 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.out‚273.3125 cpu‚26.02343861679338$ disk.nvme0n1.out‚30236.177734375 disk.sda3.in‚0 network.recv‚ 1.928545e+06& disk.nvme0c0n1.out‚30236.177734375 disk.rbd3.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd3.out‚0! disk.dm-0.out‚30236.177734375" disk.nvme0c0n1.in‚ 357.17578125 disk.drbd2165.out‚0 disk.sda3.out‚273.3125$ memory_percent‚97.34764099121094* proc.memory.percent‚16.701602935791016‰f½2Ð- _wandbruntime‚7318  _runtime‚7318kÐu˜ÇÑÁ & lr-AdamW/pg1‚0.0002963179777513788 & lr-AdamW/pg2‚0.0002963179777513788 trainer/global_step‚8599 % _timestamp‚1.7842801268337202e+09 _runtime‚7318.960481666 _step‚705·0ÌÒÆ trainer/global_step‚8599 % _timestamp‚1.7842801268337202e+09 _runtime‚7318.960481666 _step‚705 & lr-AdamW/pg1‚0.0002963179777513788 & lr-AdamW/pg2‚0.0002963179777513788Áhþÿ÷2Ó- _wandbruntime‚7318  _runtime‚7318<ËB^ô'ÔîO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9092614644736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79639366 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1315922551459840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚7318.960604788 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1306829578436608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9092955983872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1309188070579870 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1315922551459840 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1315922193081344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79639366 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1306829595475968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147822391 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68173048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9078740618652 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 % _timestamp‚1.7842801268344753e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9194270745088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68173046 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79649343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15065801 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15065798 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297139817733632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1306829595475968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9092614644736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1300108989411328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1315922193081344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18525533 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68173048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18525798 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33591599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1309187712990620 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1300108972371968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9079081168542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚706 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79649343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33591331 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269Ö~O9ù'ÕóO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1315922193081344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79639366 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1306829595475968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147822391 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68173048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9078740618652 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 % _timestamp‚1.7842801268344753e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9194270745088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68173046 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79649343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15065801 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15065798 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297139817733632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1306829595475968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9092614644736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1300108989411328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1315922193081344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18525533 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68173048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18525798 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33591599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1309187712990620 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1300108972371968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9079081168542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚706 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79649343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33591331 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9092614644736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79639366 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1315922551459840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚7318.960604788 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1306829578436608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9092955983872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1309188070579870 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1315922551459840Â$l4:2Ö- _wandbruntime‚7319  _runtime‚7319ySI þ&×øM B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287978866460672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9093627271680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15067546 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9079751672374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1316070805123584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33595047 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18527504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1306976819466240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9195282609152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1309335234716214 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1300255483043840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9093968617984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297174266737664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚7319.571883905 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147839089 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9195381664768 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68180680 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79648418 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1306976836505600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68180678 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1300255500083200 trainer/global_step‚8599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1309335592305576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79658409 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1316070446737920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33595335 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15067543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9080092222376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147839089 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9093968617984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287978885072896 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147839089 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9195381664768 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68180680 UEDeviceStatsMonitor.on_train_batch_enÝ›#ó$d/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79648418 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1306976836505600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68180678 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1300255500083200 trainer/global_step‚8599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1309335592305576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79658409 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1316070446737920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33595335 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15067543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9080092222376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147839089 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9093968617984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287978885072896 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33595047 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18527504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1306976819466240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9195282609152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1309335234716214 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1300255483043840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9093968617984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297174266737664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚7319.571883905 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9245294588416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9143310146560 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298862170216960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80103795 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1314191629918208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784280167089776e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1323334940064768 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1314191646957568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15153051 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68554646 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9143651485696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9143651485696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298862285794816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68554648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1307434505965568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1323335298443264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚289616894240768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1323335298443264 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9129697655450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚710 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1323334940064768 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1314191646957568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80103795 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979®UVÔçòO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9245391554048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33781185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1316563863071128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9129697655450 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚710 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1323334940064768 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1314191646957568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80103795 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15153048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18628134 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1307434523004928 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1314191629918208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68554646 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80093818 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9143310146560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9245294588416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9143310146560 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298862170216960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80103795 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1314191629918208 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784280167089776e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1323334940064768 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1314191646957568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15153051 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68554646 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9143651485696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9143651485696 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298862285794816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68554648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1307434505965568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1323335298443264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚289616894240768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1323335298443264 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148665148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80112861 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1314338870947840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148665148 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1316711384796722 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80102870 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148675141 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68562278 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1323483552107008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80112861 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9144664119808 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15154796 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68562280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68562278 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1314338887987200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1316711742386084 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚8649 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1314338887987200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33784986 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298896732439552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1314338870947840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1323483193721344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚289650322999296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9246409440256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9144664119808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9144322773504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148675141 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9130708709284 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9144322773504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1307581033676800 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚711 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80102870 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68562280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15154793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33784703 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18630190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚289650304387072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888ÞTeº'êýM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9246320870400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148665148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80112861 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1314338870947840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148665148 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1316711384796722 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80102870 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148675141 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68562278 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1323483552107008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80112861 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9144664119808 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15154796 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68562280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68562278 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1314338887987200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1316711742386084 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚8649 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1314338887987200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33784986 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298896732439552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1314338870947840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1323483193721344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚289650322999296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9246409440256 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9144664119808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9144322773504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148675141 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9130708709284 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9144322773504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1307581033676800 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚711 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80102870 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68562280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15154793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33784703 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18630190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚289650304387072Çß;\@2ë- _wandbruntime‚7359  _runtime‚7359LG{ ì… %loss/total_loss‚7.899529887288244 0loss/FRt2/assignment_loss‚0.8761601147197542 3loss/SRqqt2/assignment_loss‚0.49951126946166946 /loss/FRt2/detection_loss‚1.2830360030216563 1loss/SRqqt2/detection_loss‚0.6284442638079781 % _timestamp‚1.7842801677056952e+09 _runtime‚ 7359.83102378 3loss/FBt1/assignment_loss‚0.0047808921560238615 /loss/FRt1/detection_loss‚0.5022720243987581 /loss/FBt1/detection_loss‚0.5801892371153714 epoch‚21 1loss/SRqqt1/detection_loss‚0.8568576690480113 /loss/FBt2/detection_loss‚0.5675972260734332 trainer/global_step‚8649 _step‚712 0loss/FRt1/assignment_loss‚1.6401094805396088 3loss/SRqqt1/assignment_loss‚0.44823587373679036 1loss/FBt2/assignment_loss‚0.01233583320918947ç@p‹íŠ 0loss/FRt2/assignment_loss‚0.8761601147197542 3loss/SRqqt2/assignment_loss‚0.49951126946166946 /loss/FRt2/detection_loss‚1.2830360030216563 1loss/SRqqt2/detection_loss‚0.6284442638079781 % _timestamp‚1.7842801677056952e+09 _runtime‚ 7359.83102378 3loss/FBt1/assignment_loss‚0.0047808921560238615 /loss/FRt1/detection_loss‚0.5022720243987581 /loss/FBt1/detection_loss‚0.5801892371153714 epoch‚21 1loss/SRqqt1/detection_loss‚0.8568576690480113 /loss/FBt2/detection_loss‚0.5675972260734332 trainer/global_step‚8649 _step‚712 0loss/FRt1/assignment_loss‚1.6401094805396088 3loss/SRqqt1/assignment_loss‚0.44823587373679036 1loss/FBt2/assignment_loss‚0.01233583320918947 %loss/total_loss‚7.899529887288244ȯœN‰“î‚(:ˆ ìèçÒÚᓧ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üßvmúï‚(:ï ìèçÒåË°¬ disk.nvme0c0n1.in‚ 359.546875 disk.sda2.out‚0 disk.dm-1.out‚ 274.38671875# proc.memory.rssMB‚ 2787.01171875) proc.memory.percent‚17.01056957244873 network.recv‚ 1.943442e+06 disk.sda3.in‚0 disk.sda3.out‚ 274.38671875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 359.546875% disk.nvme0n1.out‚30262.1669921875 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚30262.1669921875 disk.sda.out‚ 274.38671875 network.sent‚ 1.3716237e+07 disk.drbd2165.out‚0# disk./.usageGB‚545.641773223877 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚97.22306728363037 proc.cpu.threads‚17( proc.memory.availableMB‚ 454.97265625 cpu‚26.025839471492027 disk.nvme0n1.in‚ 359.546875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚102.16455078125" disk.dm-0.out‚30262.1669921875( disk./.usagePercent‚36.6184772249963©_uQžð‚(:“ ûèçÒµµ¨¤ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ$!ýñ‚(:ò ûèçÒ¦üµ¦$ disk./.usageGB‚545.6417236328125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 274.73046875 disk.sda1.in‚0 disk.sda.in‚0) disk./.usagePercent‚36.61847389689835 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.rbd0.out‚0 network.sent‚ 1.3781312e+07 network.recv‚ 1.948995e+06% disk.nvme0n1.out‚30271.7412109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 360.26171875 disk.nvme0n1.in‚ 360.26171875" disk.nvme0c0n1.in‚ 360.26171875' proc.memory.availableMB‚ 456.1796875 proc.memory.rssMB‚2809.625* proc.memory.percent‚17.148590087890625 disk.sda3.out‚ 274.73046875' disk.nvme0c0n1.out‚30271.7412109375 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda.out‚ 274.73046875$ memory_percent‚97.21570014953613 disk.sda2.in‚0! disk.rbd1.out‚102.17626953125" disk.dm-0.out‚30271.7412109375 cpu‚26.026336014046997ª“ò‚(:ˆ ŠéçÒ±˜÷¤ gpu.0.gpu‚89.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„0ó‚(:ö ŠéçÒ–±§ disk.sda3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 361.08984375 network.sent‚ 1.3786369e+07 disk.rbd0.out‚0# disk.dm-0.out‚30278.79931640625 proc.memory.rssMB‚2809.625 disk.sda.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚102.23876953125 disk.dm-0.in‚ 361.08984375$ memory_percent‚97.21477031707764 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚36.61828726740589 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 274.95703125& disk.nvme0n1.out‚30278.79931640625$ disk./.usageGB‚545.6389427185059 disk.drbd2165.in‚0( proc.memory.availableMB‚ 456.33203125 disk.sda3.out‚ 274.95703125 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd3.in‚0 cpu‚26.027082330478347 network.recv‚ 1.952859e+06( disk.nvme0c0n1.out‚30278.79931640625 disk.rbd1.in‚0.0625* proc.memory.percent‚17.148590087890625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 361.08984375 disk.sda.out‚ 274.95703125† Ðø2ô- _wandbruntime‚7399  _runtime‚7399VázÇõÁ ' lr-AdamW/pg1‚0.00029264638424142685 ' lr-AdamW/pg2‚0.00029264638424142685 trainer/global_step‚8699 # _timestamp‚1.78428020759876e+09 _runtime‚7399.724348666 _step‚713¸iËÌöÆ ' lr-AdamW/pg1‚0.00029264638424142685 ' lr-AdamW/pg2‚0.00029264638424142685 trainer/global_step‚8699 # _timestamp‚1.78428020759876e+09 _runtime‚7399.724348666 _step‚713Éò~I²2÷- _wandbruntime‚7399  _runtime‚73999´­x+ øîO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68936246 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18731237 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80548270 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149494495 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚7399.725615469 _step‚714 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1330748045426688 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1330747687048192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9194005648384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚300584830092800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9194005648384 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1330748045426688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ÂÊÎÉZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1323940370740886 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300584714514944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9194346987520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80548270 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291288313554944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚8699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33971270 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9179973592468 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1314760056598528 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1314760039559168 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33971538 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18730972 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291288332167168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9194346987520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7842802075998192e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68936248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1321553681399808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80558247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15240301 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15240298 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33971270 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9179973592468 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1314760056598528 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1314760039559168 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33971538 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18730972 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291288332167168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9194346987520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7842802075998192e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68936248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1321553681399808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80558247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15240301 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15240298 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68936246 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18731237 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80548270 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149494495 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚7399.725615469 _step‚714 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1330748045426688 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1330747687048192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476ÊDÆF2ú- _wandbruntime‚7400  _runtime‚7400÷¤æþ&ûøM [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321700922429440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9297517605888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15242046 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9297424841728 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚715 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9195359621632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9195359621632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68943878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1324087534877230 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1321700939468800 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149511193 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1314906550231040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68943878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1330896299090432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300619167155200 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321700922429440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80567313 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1321700939468800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300619278531584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149511193 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80557322 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15242043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33975037 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18732994 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9180984646190 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291321760925696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1324087892466592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149501200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149501200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149511193 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1314906550231040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68943878 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1330896299090432 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300619167155200 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321700922429440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80567313 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1321700939468800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300619278531584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149511193 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80557322 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15242043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33975037 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18732994 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9180984646190 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291321760925696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1324087892466592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149501200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149501200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚715 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976Ë¿s\2ý- _wandbruntime‚7400  _runtime‚7400g³Ã± þ† % _timestamp‚1.7842802082134428e+09 _step‚716 2loss/FBt1/assignment_loss‚0.013921473958669883 4loss/FBt2/assignment_loss‚0.00034736294407338163 /loss/FRt1/detection_loss‚0.5040868194123352 /loss/FBt1/detection_loss‚0.5689417398189107 0loss/FRt2/assignment_loss‚0.8567063630804339 2loss/SRqqt1/assignment_loss‚0.5247209543901572 /loss/FRt2/detection_loss‚1.2104258439922664 1loss/SRqqt2/detection_loss‚0.6196517499667481 %loss/total_loss‚8.011502623013918 _runtime‚7400.338772964 2loss/SRqqt2/assignment_loss‚0.5229422139242049 1loss/SRqqt1/detection_loss‚0.8728696512818641 trainer/global_step‚8699 0loss/FRt1/assignment_loss‚1.6758500708296262 /loss/FBt2/detection_loss‚0.6410383794146279 epoch‚21«Sy¬ÿ‹ 2loss/FBt1/assignment_loss‚0.013921473958669883 4loss/FBt2/assignment_loss‚0.00034736294407338163 /loss/FRt1/detection_loss‚0.5040868194123352 /loss/FBt1/detection_loss‚0.5689417398189107 0loss/FRt2/assignment_loss‚0.8567063630804339 2loss/SRqqt1/assignment_loss‚0.5247209543901572 /loss/FRt2/detection_loss‚1.2104258439922664 1loss/SRqqt2/detection_loss‚0.6196517499667481 %loss/total_loss‚8.011502623013918 _runtime‚7400.338772964 2loss/SRqqt2/assignment_loss‚0.5229422139242049 1loss/SRqqt1/detection_loss‚0.8728696512818641 trainer/global_step‚8699 0loss/FRt1/assignment_loss‚1.6758500708296262 /loss/FBt2/detection_loss‚0.6410383794146279 epoch‚21 % _timestamp‚1.7842802082134428e+09 _step‚716Ìa“€‚(:ˆ ™éçÒÈ«®¥ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚245.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.0762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°Ñ ½è‚(:Ý ™éçÒ‡óæ§) disk./.usagePercent‚36.61862775742643 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 30288.046875 disk.sda.in‚0 disk.sda3.out‚ 275.203125$ memory_percent‚97.21474647521973 proc.memory.rssMB‚2809.625 disk.sda2.out‚0 disk.rbd3.out‚0* proc.memory.percent‚17.148590087890625 proc.cpu.threads‚17! disk.nvme0n1.out‚ 30288.046875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 275.203125 disk.dm-0.out‚ 30288.046875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 361.8046875! disk.rbd1.out‚102.23876953125 network.sent‚ 1.3802433e+07$ disk./.usageGB‚545.6440162658691' proc.memory.availableMB‚ 456.3359375 disk.drbd2165.in‚0 disk.sda.out‚ 275.203125 disk.sda3.in‚0 disk.rbd3.in‚0 cpu‚26.027661847886872 disk.rbd0.in‚0 network.recv‚ 1.956416e+06 disk.nvme0n1.in‚ 361.8046875 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 361.8046875Gf!‘‚‚(:† ¨éçÒ…¿¢ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚203.92' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¾)ùƒ‚(:î ¨éçÒçäí¦$ disk./.usageGB‚545.6441078186035& disk.nvme0n1.out‚30296.29150390625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 275.3671875$ memory_percent‚97.21436500549316' proc.memory.availableMB‚ 456.3984375( disk.nvme0c0n1.out‚30296.29150390625 disk.sda.out‚ 275.3671875 disk.rbd3.out‚0 proc.memory.rssMB‚2809.625 network.sent‚ 1.3854419e+07 disk.sda3.in‚0* proc.memory.percent‚17.148590087890625* disk./.usagePercent‚36.618633901607254 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 362.6171875 disk.sda2.in‚0 disk.rbd1.in‚0.0625 network.recv‚ 1.96137e+06 disk.sda2.out‚0# disk.dm-0.out‚30296.29150390625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 275.3671875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 362.6171875 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 362.6171875 cpu‚26.028349465963192 disk.rbd0.out‚0 disk.rbd1.out‚102.3564453125–ãûh‘„‚(:† ·éçÒžÂ㤠gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚256.11' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÐ:Áô…‚(:é ·éçÒˆ×û¦ network.recv‚ 1.964435e+06 disk.dm-1.out‚ 276.09375( proc.memory.availableMB‚ 456.31640625 proc.cpu.threads‚17 disk.sda.out‚ 276.09375) disk./.usagePercent‚36.61863620567506 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚30303.55224609375 network.sent‚ 1.3857276e+07$ disk./.usageGB‚545.6441421508789 disk.sda3.in‚0 disk.sda3.out‚ 276.09375 disk.nvme0n1.in‚ 362.6171875$ memory_percent‚97.21486568450928 proc.memory.rssMB‚2809.625* proc.memory.percent‚17.148590087890625 disk.dm-1.in‚0 disk.rbd1.out‚102.3564453125 disk.drbd2165.in‚0 disk.sda.in‚0 cpu‚26.028959785960865 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 362.6171875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0# disk.dm-0.out‚30303.55224609375& disk.nvme0n1.out‚30303.55224609375 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 362.6171875G)ô2†- _wandbruntime‚7440  _runtime‚7440ï5bLJÁ _runtime‚7440.029535837 _step‚717 & lr-AdamW/pg1‚0.0002908051032230703 & lr-AdamW/pg2‚0.0002908051032230703 trainer/global_step‚8749 % _timestamp‚1.7842802479039564e+09Å=_pÌˆÆ & lr-AdamW/pg1‚0.0002908051032230703 & lr-AdamW/pg2‚0.0002908051032230703 trainer/global_step‚8749 % _timestamp‚1.7842802479039564e+09 _runtime‚7440.029535837 _step‚717Í¢…y:2‰- _wandbruntime‚7440  _runtime‚7440ÙÉô'ŠîO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1328915749920768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230590079376 _runtime‚7440.030779304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81012699 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81002722 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34161163 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842802479049897e+09 _step‚718 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1322085590192128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1322085573152768 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9245042489344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292959770093568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1338160434031616 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1338160792410112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1328915732881408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81002722 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9230930629266 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150320568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚8749 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1338160792410112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1328915732881408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15327548 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1331316163232144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69317846 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69317846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9244701150208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69317848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34161422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292959751481344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9347470891520 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150330547 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302307222372864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Kräñù'‹óO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69317846 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69317846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9244701150208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69317848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34161422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292959751481344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9347470891520 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150330547 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302307222372864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1328915749920768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230590079376 _runtime‚7440.030779304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81012699 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81002722 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34161163 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842802479049897e+09 _step‚718 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1322085590192128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1322085573152768 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9245042489344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292959770093568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1338160434031616 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1338160792410112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1328915732881408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81002722 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9230930629266 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150320568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚8749 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1338160792410112 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1328915732881408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15327548 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1331316163232144Î>Òå2Œ- _wandbruntime‚7440  _runtime‚7440°Œt{øM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚719 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150337252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69325478 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34164990 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1329062973911040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚292993180239872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150337252 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81011774 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚292993198852096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9348577731584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚7440.830828284 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18835965 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302341776583680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842802487050226e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1322232100864000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1331463684957738 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9231941683100 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150347245 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81021765 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81021765 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9348487064576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69325478 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1338309046073856 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1329062990950400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150347245 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69325480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1338309046073856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9245713777152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8749 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302341667304448 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1322232083824640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69325480 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1329062973911040 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1338308687688192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Mcwº'ŽýM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9245713777152 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81021765 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81021765 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9348487064576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69325478 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1338309046073856 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1329062990950400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150347245 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69325480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1338309046073856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9245713777152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8749 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302341667304448 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1322232083824640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69325480 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1329062973911040 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1338308687688192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚719 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150337252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69325478 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34164990 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1329062973911040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚292993180239872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150337252 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81011774 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚292993198852096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9348577731584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚7440.830828284 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18835965 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302341776583680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842802487050226e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1322232100864000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1331463684957738 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9231941683100 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150347245 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1345573181015040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1345573539393536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9295737991168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15414798 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15414801 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9281206566284 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1345573539393536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚294631208019968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1336277801402368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9295396652032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9295396652032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34351294 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34351035 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294631189407744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9295737991168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69699446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151156620 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81467151 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1345573181015040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚8799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69699448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18936493 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69699446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚†Ìé‚V0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1338692313312652 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 $ _timestamp‚1.784280288356776e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1345573181015040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1345573539393536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9295737991168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15414798 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15414801 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9281206566284 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1345573539393536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚294631208019968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1336277801402368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9295396652032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9295396652032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34351294 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34351035 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚294631189407744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9295737991168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69699446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151156620 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81467151 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1345573181015040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚8799ÒR‰³ô2œ- _wandbruntime‚7481  _runtime‚7481ÝYs÷þ&øM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚723 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9296750625280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9296409278976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15416546 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304064169784832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151183297 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304064060505600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34354853 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1345721793057280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9296409278976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294664618166272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151173304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9296750625280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81466226 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69707078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15416543 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9399442339328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9282217620006 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1336425025392640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151183297 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294664636778496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1336425042432000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18938310 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69707078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18938578 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1336425042432000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1336425025392640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81476217 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1345721793057280 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1345721434671616àQ*'žýM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1336425042432000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18938310 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69707078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18938578 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1336425042432000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1336425025392640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81476217 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1345721793057280 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1345721434671616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚723 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9296750625280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9296409278976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15416546 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304064169784832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151183297 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304064060505600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34354853 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1345721793057280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9296409278976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294664618166272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151173304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9296750625280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81466226 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69707078 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15416543 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9399442339328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9282217620006 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1336425025392640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151183297 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294664636778496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 Eí2/assignment_loss‚0.5362814087944274 1loss/FBt1/assignment_loss‚0.02039387742496954 1loss/SRqqt1/detection_loss‚0.8877258965329937 /loss/FBt1/detection_loss‚0.4228261717394224 .loss/FBt2/detection_loss‚0.477255600896713•p(Ÿ¡ˆ 0loss/FRt2/assignment_loss‚0.8460673136987548 3loss/SRqqt1/assignment_loss‚0.41262605216994896 /loss/FRt1/detection_loss‚0.5049014796956839 /loss/FRt2/detection_loss‚1.2947581444625584 %loss/total_loss‚7.681346774703249 0loss/FRt1/assignment_loss‚1.5954740100202307 2loss/SRqqt2/assignment_loss‚0.5362814087944274 1loss/FBt1/assignment_loss‚0.02039387742496954 1loss/SRqqt1/detection_loss‚0.8877258965329937 /loss/FBt1/detection_loss‚0.4228261717394224 .loss/FBt2/detection_loss‚0.477255600896713 epoch‚21 % _timestamp‚1.7842802889715412e+09 2loss/FBt2/assignment_loss‚0.015575864201571333 1loss/SRqqt2/detection_loss‚0.6674609550659742 trainer/global_step‚8799 _runtime‚7481.096878961 _step‚724Ô—^¥·¢‚(:’ äéçÒ‹¿®¥ gpu.0.gpu‚64.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚290.855' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.17100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IëwLó£‚(:è äéçÒí£«¨& disk.nvme0n1.out‚30327.98779296875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.out‚102.5439453125 disk.dm-0.in‚364.875 cpu‚26.032461771236644 network.recv‚ 1.976663e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 277.0234375 disk.sda2.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.21729755401611 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚30327.98779296875# proc.memory.rssMB‚ 2809.83203125* proc.memory.percent‚17.149853706359863 network.sent‚ 1.3929671e+07) disk./.usagePercent‚36.61865515023259 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚364.875 disk.nvme0c0n1.in‚364.875 disk.dm-1.out‚ 277.0234375 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 277.0234375# disk.dm-0.out‚30327.98779296875( proc.memory.availableMB‚ 455.91796875$ disk./.usageGB‚545.6444244384766Õj$22¤- _wandbruntime‚7499  _runtime‚7499¢ÌŽ¾=¥¸{ =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24164393544197083 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6499123573303223 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9836840033531189 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10360997170209885 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26466184854507446 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03680998086929321 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8660644292831421 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8660644292831421 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894620776176453 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4349517226219177 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9946548938751221 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2358151376247406 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7179365158081055 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9971964955329895 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0465000756084919 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6666666269302368 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3133775591850281 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3162052631378174 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2769790291786194 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5004133582115173 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_1_of_5‚1 trainer/global_step‚8799 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944220185279846 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08420275896787643 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9937242865562439 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2561839818954468 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_4‚0.8069139122962952 particle/accuracy_2_of_3‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 *jet/accuracy_1_of_3‚0.9330746531486511 epoch‚21 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29442495107650757 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6336090564727783 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5033705234527588 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9259598255157471 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6086471676826477 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0362238734960556 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17083331942558289 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5920262932777405 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5773619413375854 7"Purity/*FRt2SRqqt*FBt/event_purity‚0.23143370449543 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.710206151008606 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.654137134552002 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04152138903737068 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7345010042190552 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.033636245876550674 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7169824838638306 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7599261403083801 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954078197479248 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5144301652908325 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7101989388465881 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8127928972244263 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18753759562969208 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_3‚0.07363613694906235 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6988032460212708 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6458483338356018 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 _step‚725 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9246812462806702 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.532997727394104 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2728996276855469 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.026251083239912987 +particle/sensitivity‚0.6193592548370361 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958121180534363 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 particle/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03322882205247879 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2292344719171524 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_1‚0.3573537766933441 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.40625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7102230191230774 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7920351624488831 % _timestamp‚1.7842803069367354e+09 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6772947907447815 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6794928312301636 -"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26210442185401917 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3851044476032257 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026251083239912987 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.658829927444458 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920517206192017 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1724260002374649 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961851239204407 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9977324604988098 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9902719855308533 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7408013343811035 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.9671030640602112 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05447806790471077 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2598218321800232 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6423161029815674 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30366241931915283 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7245985865592957 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08789979666471481 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚8.114115805367797 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3833881914615631 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6381755471229553 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053256839513778687 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9903001189231873 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8547300696372986 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7873142957687378 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6353971362113953 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5863754749298096 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35660097002983093 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9908525347709656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5051978826522827 *jet/accuracy_1_of_2‚0.8159002661705017 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9035318493843079 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7252203226089478 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8069917559623718 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950188398361206 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7387586236000061 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8216364979743958 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 5validation_average_jet_accuracy‚0.396980881690979 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 'particle/f_score‚0.6352216005325317 jet/accuracy_5_of_5‚0 _runtime‚7499.063184011 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2710079252719879 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3278225362300873 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6458483338356018 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31388598680496216 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.238407701253891 /particle/accuracy_4_of_4‚0.9467267394065857 particle/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997686505317688 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30366241931915283 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7770597338676453 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7443997263908386 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7128489017486572 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9918419718742371 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2940491735935211 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30306851863861084 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974292516708374 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7863476872444153 +particle/specificity‚0.9095739722251892 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5529167056083679 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9924123883247375 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6589636206626892 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.863984227180481 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2004888653755188 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9939866662025452 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6543967127799988 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31154370307922363 (particle/accuracy‚0.8472493290901184 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6760922074317932 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_2_of_2‚0.16288425028324127 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6952359080314636 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18549863994121552 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5295037031173706 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8212774991989136 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7661299705505371 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7874656915664673 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5889984369277954 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6419727206230164 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3609500229358673 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31386086344718933 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974292516708374Dn°T8¦½{ ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5529167056083679 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9924123883247375 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6589636206626892 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.863984227180481 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2004888653755188 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9939866662025452 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6543967127799988 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31154370307922363 (particle/accuracy‚0.8472493290901184 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6760922074317932 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_2_of_2‚0.16288425028324127 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6952359080314636 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18549863994121552 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5295037031173706 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8212774991989136 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7661299705505371 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7874656915664673 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5889984369277954 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6419727206230164 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3609500229358673 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31386086344718933 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974292516708374 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24164393544197083 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6499123573303223 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9836840033531189 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10360997170209885 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26466184854507446 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03680998086929321 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8660644292831421 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8660644292831421 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894620776176453 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4349517226219177 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9946548938751221 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2358151376247406 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7179365158081055 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9971964955329895 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0465000756084919 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6666666269302368 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3133775591850281 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3162052631378174 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2769790291786194 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5004133582115173 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_1_of_5‚1 trainer/global_step‚8799 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944220185279846 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08420275896787643 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9937242865562439 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2561839818954468 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_4‚0.8069139122962952 particle/accuracy_2_of_3‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 *jet/accuracy_1_of_3‚0.9330746531486511 epoch‚21 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29442495107650757 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6336090564727783 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5033705234527588 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9259598255157471 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6086471676826477 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0362238734960556 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17083331942558289 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5920262932777405 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5773619413375854 7"Purity/*FRt2SRqqt*FBt/event_purity‚0.23143370449543 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.710206151008606 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.654137134552002 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04152138903737068 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7345010042190552 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.033636245876550674 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7169824838638306 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7599261403083801 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954078197479248 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5144301652908325 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7101989388465881 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8127928972244263 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18753759562969208 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_3‚0.07363613694906235 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6988032460212708 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6458483338356018 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 _step‚725 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9246812462806702 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.532997727394104 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2728996276855469 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.026251083239912987 +particle/sensitivity‚0.6193592548370361 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958121180534363 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 particle/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03322882205247879 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2292344719171524 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_1‚0.3573537766933441 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.40625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7102230191230774 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7920351624488831 % _timestamp‚1.7842803069367354e+09 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6772947907447815 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6794928312301636 -"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26210442185401917 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3851044476032257 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026251083239912987 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.658829927444458 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920517206192017 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1724260002374649 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961851239204407 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9977324604988098 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9902719855308533 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7408013343811035 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.9671030640602112 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05447806790471077 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2598218321800232 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6423161029815674 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30366241931915283 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7245985865592957 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08789979666471481 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚8.114115805367797 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3833881914615631 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6381755471229553 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053256839513778687 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9903001189231873 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8547300696372986 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7873142957687378 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6353971362113953 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5863754749298096 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35660097002983093 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9908525347709656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5051978826522827 *jet/accuracy_1_of_2‚0.8159002661705017 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9035318493843079 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7252203226089478 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8069917559623718 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950188398361206 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7387586236000061 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8216364979743958 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 5validation_average_jet_accuracy‚0.396980881690979 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 'particle/f_score‚0.6352216005325317 jet/accuracy_5_of_5‚0 _runtime‚7499.063184011 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2710079252719879 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3278225362300873 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6458483338356018 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31388598680496216 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.238407701253891 /particle/accuracy_4_of_4‚0.9467267394065857 particle/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997686505317688 >&Puri1Ôoty/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30366241931915283 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7770597338676453 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7443997263908386 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7128489017486572 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9918419718742371 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2940491735935211 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30306851863861084 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974292516708374 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7863476872444153 +particle/specificity‚0.9095739722251892Õ®ý§‚(:’ óéçÒà”¶¤ gpu.0.gpu‚91.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿs3í¨‚(:â óéçÒÈ­Õ¦ disk.nvme0n1.in‚364.875( proc.memory.availableMB‚ 455.83984375 network.sent‚ 1.4005484e+07 network.recv‚ 1.981621e+06 disk.sda.in‚0 disk.sda.out‚ 277.51171875 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚102.5439453125 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-0.out‚30333.50390625 disk.sda1.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.out‚ 277.51171875 disk.dm-1.out‚ 277.51171875# disk.nvme0n1.out‚30333.50390625$ memory_percent‚97.21777439117432) disk./.usagePercent‚36.61864644597643 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚364.875# proc.memory.rssMB‚ 2809.83203125 cpu‚26.039665422098533$ disk./.usageGB‚545.6442947387695 disk.nvme0c0n1.in‚364.875% disk.nvme0c0n1.out‚30333.50390625 disk.drbd2165.out‚0* proc.memory.percent‚17.149853706359863G}Г©‚(:ˆ ‚êçÒùÍÏ¢ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý­ˆè骂(:Þ ‚êçÒÕœ‹§) disk./.usagePercent‚36.61863748571273$ disk./.usageGB‚545.6441612243652 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 365.703125( proc.memory.availableMB‚ 455.41796875 network.recv‚ 1.986483e+06 disk.sda3.out‚277.8125 cpu‚26.04017870659817 proc.cpu.threads‚17 disk.nvme0n1.in‚ 365.703125% disk.nvme0c0n1.out‚30341.33984375# proc.memory.rssMB‚ 2809.83203125 disk.dm-0.out‚30341.33984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚277.8125 disk.rbd1.out‚102.6064453125 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚277.8125# disk.nvme0n1.out‚30341.33984375 disk.sda3.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.149853706359863 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.22034931182861 disk.dm-0.in‚ 365.703125 network.sent‚ 1.4052121e+07¥(Í'“«‚(:ˆ ‘êçÒ冷¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÄ×Ь‚(:÷ ‘êçÒćХ disk.rbd3.out‚0 network.recv‚ 1.990178e+06$ disk./.usageGB‚545.6440811157227 disk.dm-0.in‚ 366.41796875( disk.nvme0c0n1.out‚30352.37646484375 disk.sda.out‚ 278.01171875 disk.rbd1.out‚102.6064453125 disk.rbd3.in‚0 cpu‚26.040769756098612) disk./.usagePercent‚36.61863210955451 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 366.41796875# memory_percent‚97.2203254699707# proc.memory.rssMB‚ 2809.83203125* proc.memory.percent‚17.149853706359863 disk.drbd2165.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 366.41796875 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.sda.in‚0 disk.sda3.in‚0 network.sent‚ 1.4057147e+07 disk.sda1.out‚0 disk.sda3.out‚ 278.01171875& proc.memory.availableMB‚ 455.421875# disk.dm-0.out‚30352.37646484375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 278.01171875& disk.nvme0n1.out‚30352.37646484375 disk.sda2.out‚0®Æ¨12­- _wandbruntime‚7539  _runtime‚7539¡|nâÇ®Á & lr-AdamW/pg1‚0.0002871121639621956 & lr-AdamW/pg2‚0.0002871121639621956 trainer/global_step‚8849 % _timestamp‚1.7842803469547164e+09 _runtime‚7539.080399991 _step‚726ij‡Ì¯Æ & lr-AdamW/pg1‚0.0002871121639621956 & lr-AdamW/pg2‚0.0002871121639621956 trainer/global_step‚8849 % _timestamp‚1.7842803469547164e+09 _runtime‚7539.080399991 _step‚726ÖQî[×2°- _wandbruntime‚7539  _runtime‚7539qò'±ìO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70201712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 $ _timestamp‚1.784280346955841e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9363825359360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9467809720320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9364166698496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296318558046208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1338645303554048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9364166698496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1354933227854336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70201714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81981948 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305786447954944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9467889908736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15504568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70201712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1345569419534336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1354933586232832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9349883328311 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81991925 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9363825359360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70201714 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1354933227854336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 trainer/global_step‚8849 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1354933586232832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296318539433984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1347994846332469 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9349542778421 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81991925 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1345569402494976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70201712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 $ _timestamp‚1.784280346955841e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9363825359360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9467809720320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9364166698496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296318558046208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1338645303554048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9364166698496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1354933227854336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70201714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81981948 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305786447954944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9467889908736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15504568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70201712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1345569419534336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1354933586232832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9349883328311 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81991925 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9363825359360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70201714 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1354933227854336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 trainer/global_step‚8849 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1354933586232832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296318539433984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1347994846332469 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9349542778421 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81991925×%™Â2³- _wandbruntime‚7539  _runtime‚7539ŸÎ{´÷M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152210337 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1355081481510912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15506313 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1355081839896576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70209344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34581037 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1345716660563968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152200344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70209346 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305820808413696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296351986804736 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1338791814225920 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82000991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceyFF‚StatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1355081839896576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9468840221184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842803475685492e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1345716643524608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81991000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34580780 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1338791831265280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1348142725647425 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152210337 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81991000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82000991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70209344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305820907207168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚7539.694461708 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9364837986304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1355081481510912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15506313 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1355081839896576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70209344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34581037 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1345716660563968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152200344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70209346 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305820808413696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296351986804736 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1338791814225920 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82000991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1355081839896576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9468840221184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842803475685492e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1345716643524608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81991000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34580780 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1338791831265280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1348142725647425 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152210337 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81991000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82000991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70209344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305820907207168 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚7539.694461708 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9364837986304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152210337 E©ý¯“»‚(:ˆ ¯êçÒÏ£í£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÞI'¼‚(:õ ¯êçÒ”¥ß¥ disk.dm-0.in‚ 367.9609375 disk.sda1.out‚0$ memory_percent‚97.22018241882324* disk./.usagePercent‚36.618653870194926 disk.rbd0.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2809.83203125* proc.memory.percent‚17.149853706359863 disk.sda3.out‚ 278.55078125 network.recv‚ 1.998929e+06 disk.rbd0.in‚0 disk.sda.out‚ 278.55078125 disk.drbd2165.in‚0 network.sent‚ 1.41252e+07 disk.sda.in‚0 cpu‚26.041955797286583 proc.cpu.threads‚17 disk.rbd1.in‚0.0625# disk.dm-0.out‚30370.25341796875& disk.nvme0n1.out‚30370.25341796875( disk.nvme0c0n1.out‚30370.25341796875 disk.sda3.in‚0 disk.nvme0n1.in‚ 367.9609375 disk.drbd2165.out‚0$ disk./.usageGB‚545.6444053649902 disk.sda2.in‚0 disk.rbd1.out‚102.6689453125 disk.rbd3.out‚0' proc.memory.availableMB‚ 455.4453125! disk.nvme0c0n1.in‚ 367.9609375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 278.55078125 disk.rbd3.in‚0ÉD¾ù½‚(:’ ¾êçÒ„±ý¤ gpu.0.gpu‚75.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.921' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.38419999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èñ¨ˆþ¾‚(:ó ¾êçÒ‘…›§# disk./.usageGB‚545.644474029541 disk.dm-1.in‚0# disk.dm-0.out‚30373.53662109375 disk.nvme0n1.in‚ 367.9609375 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 455.640625 network.sent‚ 1.4130308e+07! disk.nvme0c0n1.in‚ 367.9609375$ memory_percent‚97.21899032592773* proc.memory.percent‚17.149853706359863 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚ 2.002327e+06 disk.rbd1.in‚0.0625 disk.rbd1.out‚102.7314453125 disk.sda1.out‚0# proc.memory.rssMB‚ 2809.83203125) disk./.usagePercent‚36.61865847833054 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚26.04264998057658 disk.dm-1.out‚ 278.94140625 disk.dm-0.in‚ 367.9609375 disk.rbd3.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30373.53662109375 disk.sda.out‚ 278.94140625 disk.sda3.in‚0 disk.sda3.out‚ 278.94140625 disk.rbd3.out‚0& disk.nvme0n1.out‚30373.53662109375 proc.cpu.threads‚17 ƒC2¿- _wandbruntime‚7579  _runtime‚7579höÝÉÀà % _timestamp‚1.7842803875927327e+09 _runtime‚7579.718413401 _step‚730 ' lr-AdamW/pg1‚0.00028526074291026544 ' lr-AdamW/pg2‚0.00028526074291026544 trainer/global_step‚8899:øÎÁÈ ' lr-AdamW/pg1‚0.00028526074291026544 ' lr-AdamW/pg2‚0.00028526074291026544 trainer/global_step‚8899 % _timestamp‚1.7842803875927327e+09 _runtime‚7579.718413401 _step‚730ÚxnÒ2Â- _wandbruntime‚7579  _runtime‚7579×Öêó'ÃíO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8899 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82436400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _runtime‚7579.723367563 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70583314 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9414862200320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297989977360384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9400499815219 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307509413838848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1362345974837760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1362345974837760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19177356 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307509315038208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9414520861184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297989995972608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9414520861184 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15591818 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1355371354002227 _step‚731 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34769174 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1352931471015936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82436400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70583314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9519417866240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1352931453976576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9400159265329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1362346333216256 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1345970854187008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153019712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1352931471015936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1352931453976576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34769440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9519337677824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19177619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1355370996412977 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82446377 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70583312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9414862200320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70583312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15591821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263¯ôx7ÄòO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82436400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70583314 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9519417866240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1352931453976576 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9400159265329 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1362346333216256 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1345970854187008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153019712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1352931471015936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1352931453976576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34769440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9519337677824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=Devicx;ïÂõeStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19177619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1355370996412977 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82446377 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70583312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9414862200320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70583312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15591821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8899 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82436400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _runtime‚7579.723367563 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70583314 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9414862200320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297989977360384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9400499815219 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307509413838848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1362345974837760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1362345974837760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19177356 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307509315038208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9414520861184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297989995972608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9414520861184 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15591818 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1355371354002227 _step‚731 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34769174 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1352931471015936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9415874834432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1355518875727933 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _runtime‚7580.330989535 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9520448840704 _step‚732 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19179429 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9415533488128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9520374950912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842803882052183e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1353078695006208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70590946 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153046389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82455443 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1353078712045568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9401510869053 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82445452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70590944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9401170319051 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298023424731136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9415533488128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9415874834432 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153036396 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34772992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19179690 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269óX£æ'ÇüM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9520448840704 _step‚732 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19179429 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9415533488128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9520374950912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842803882052183e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1353078695006208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70590946 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153046389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82455443 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1353078712045568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9401510869053 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82445452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70590944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9401170319051 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298023424731136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9415533488128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9415874834432 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153036396 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34772992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19179690 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153046389 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82445452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9415874834432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1355518875727933 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _runtime‚7580.330989535Ü°72È- _wandbruntime‚7580  _runtime‚7580x_\ Ɇ /loss/FRt1/assignment_loss‚1.624755732785492 2loss/SRqqt1/assignment_loss‚0.5748321016522938 4loss/FBt1/assignment_loss‚0.00021059152268441362 1loss/FBt2/assignment_loss‚0.07338982187187992 1loss/SRqqt2/detection_loss‚0.6430788964981787 &loss/total_loss‚7.7618575754248464 epoch‚22 _step‚733 /loss/FBt2/detection_loss‚0.4266674759004403 % _timestamp‚1.7842803882081976e+09 0loss/FRt2/assignment_loss‚0.8010959099797369 /loss/FRt1/detection_loss‚0.5162479799867843 1loss/SRqqt1/detection_loss‚0.9150641689559246 trainer/global_step‚8899 _runtime‚7580.333536403 3loss/SRqqt2/assignment_loss‚0.49403112851596587 /loss/FRt2/detection_loss‚1.2738131450364836 /loss/FBt1/detection_loss‚0.4186706227189823ÌFp¬Ê‹ _runtime‚7580.333536403 3loss/SRqqt2/assignment_loss‚0.49403112851596587 /loss/FRt2/detection_loss‚1.2738131450364836 /loss/FBt1/detection_loss‚0.4186706227189823 /loss/FRt1/assignment_loss‚1.624755732785492 2loss/SRqqt1/assignment_loss‚0.5748321016522938 4loss/FBt1/assignment_loss‚0.00021059152268441362 1loss/FBt2/assignment_loss‚0.07338982187187992 1loss/SRqqt2/detection_loss‚0.6430788964981787 &loss/total_loss‚7.7618575754248464 epoch‚22 _step‚733 /loss/FBt2/detection_loss‚0.4266674759004403 % _timestamp‚1.7842803882081976e+09 0loss/FRt2/assignment_loss‚0.8010959099797369 /loss/FRt1/detection_loss‚0.5162479799867843 1loss/SRqqt1/detection_loss‚0.9150641689559246 trainer/global_step‚8899ÝUC§ã“Ë‚(:ˆ ÍêçÒÉäठgpu.0.gpu‚94.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-5ªŽõÌ‚(:ê ÍêçÒçÈô¦! disk.nvme0c0n1.in‚ 368.7890625 disk.sda3.out‚279.1875 disk.nvme0n1.in‚ 368.7890625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚545.6445846557617 disk.dm-1.out‚279.1875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚30383.42529296875 disk.sda1.out‚0( proc.memory.availableMB‚ 455.61328125 disk.sda2.in‚0 disk.dm-1.in‚0 network.sent‚ 1.4133373e+07 disk.sda2.out‚0 disk.rbd1.out‚102.7314453125 disk.dm-0.in‚ 368.7890625 network.recv‚ 2.006278e+06 disk.sda.out‚279.1875 disk.sda3.in‚0# memory_percent‚97.2191572189331* proc.memory.percent‚17.149853706359863 proc.cpu.threads‚17) disk./.usagePercent‚36.61866590254903 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30383.42529296875 disk.sda.in‚0# disk.dm-0.out‚30383.42529296875 disk.sda1.in‚0# proc.memory.rssMB‚ 2809.83203125 cpu‚26.043190454550054p„“Í‚(:ˆ ÜêçÒÛËѤ gpu.0.gpu‚71.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽæG×á΂(:Ö ÜêçÒ¯‚Ú¦$ disk./.usageGB‚545.6444244384766! disk.dm-0.out‚30390.083984375& disk.nvme0c0n1.out‚30390.083984375 disk.sda.out‚ 279.515625 disk.nvme0n1.in‚369.5 disk.sda1.out‚0# memory_percent‚97.2203254699707& proc.memory.availableMB‚ 455.421875 proc.cpu.threads‚17 network.sent‚ 1.4198457e+07 disk.sda3.in‚0 disk.rbd1.out‚102.8564453125 disk.dm-0.in‚369.5 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2809.83203125 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0) disk./.usagePercent‚36.61865515023259 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 279.515625 disk.rbd3.out‚0$ disk.nvme0n1.out‚30390.083984375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.149853706359863 disk.sda3.out‚ 279.515625 disk.sda1.in‚0 cpu‚26.043793500313818 network.recv‚ 2.011463e+06 disk.nvme0c0n1.in‚369.5 disk.dm-1.in‚0jkŸ—;Ï‚(:† ëêçÒٰߤ gpu.0.gpu‚95.0 gpu.0.memo=gÛAVry‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œº&üЂ(:ñ ëêçÒãÿñ¦ disk.drbd2165.in‚0$ memory_percent‚97.21672534942627 proc.cpu.threads‚17$ disk./.usageGB‚545.6438941955566 disk.dm-0.in‚ 370.328125# disk.dm-0.out‚30399.38720703125* proc.memory.percent‚17.149853706359863 network.sent‚ 1.4203475e+07* disk./.usagePercent‚36.618619565185334 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 456.01171875 disk.rbd0.out‚0 network.recv‚ 2.015126e+06( disk.nvme0c0n1.out‚30399.38720703125 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 279.7265625 disk.dm-1.out‚ 279.7265625 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.rbd1.out‚102.8564453125 disk.nvme0n1.in‚ 370.328125 cpu‚26.04435873275546 disk.nvme0c0n1.in‚ 370.328125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚30399.38720703125 disk.drbd2165.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2809.83203125 disk.sda.out‚ 279.7265625 disk.sda1.in‚0Xðþ2Ñ- _wandbruntime‚7620  _runtime‚7620V“ÞXÉÒà ' lr-AdamW/pg1‚0.00028340617939193527 ' lr-AdamW/pg2‚0.00028340617939193527 trainer/global_step‚8949 % _timestamp‚1.7842804279197266e+09 _runtime‚7620.045278564 _step‚734¡æ“ÐÎÓÈ % _timestamp‚1.7842804279197266e+09 _runtime‚7620.045278564 _step‚734 ' lr-AdamW/pg1‚0.00028340617939193527 ' lr-AdamW/pg2‚0.00028340617939193527 trainer/global_step‚8949ÞÄiU^2Ô- _wandbruntime‚7620  _runtime‚7620ÛoÝ™ó'ÕíO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153865743 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82900829 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34961173 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9465216363008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309232130576896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82900829 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15679071 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚299661415286784 _runtime‚ 7620.04657368 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82890852 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1369758721821184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1360293522497536 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1369759080199680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19282102 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1360293505458176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9465557702144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309232246154752 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70964912 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1369759080199680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70964914 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9465216363008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1360293505458176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 trainer/global_step‚8949 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1369758721821184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1362747146493485 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70964914 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299661433899008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0œl—}ø'ÖòO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1360293522497536 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1369759080199680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19282102 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1360293505458176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9465557702144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309232246154752 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70964912 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1369759080199680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70964914 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9465216363008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1360293505458176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 trainer/global_step‚8949 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1369758721821184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1362747146493485 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70964914 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299661433899008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153865743 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82900829 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34961173 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9465216363008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309232130576896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82900829 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15679071 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚299661415286784 _runtime‚ 7620.04657368 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82890852 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1369758721821184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376ß°Ì2×- _wandbruntime‚7620  _runtime‚7620;:ü&ØöM XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1369907333863424 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1369906975477760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34964987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309266695836672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9571833179136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9466570336256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34964731 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚7620.661425529 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19283918 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299694844045312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153872448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15680813 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153882441 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299694862657536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82909895 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82899904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1362895025808441 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9466570336256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70972544 trainer/global_step‚8949äí2ÙMÙûM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9466570336256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34964731 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚7620.661425529 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19283918 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299694844045312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153872448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15680813 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153882441 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299694862657536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82909895 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82899904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1362895025808441 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9466570336256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70972544 trainer/global_step‚8949 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1369907333863424 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1369906975477760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34964987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309266695836672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9571833179136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362à2ê€j2Ú- _wandbruntime‚7620  _runtime‚762036b Û‚ trainer/global_step‚8949 _runtime‚7620.663016334 /loss/FRt1/assignment_loss‚1.673819718727899 2loss/SRqqt1/assignment_loss‚0.5160462643538873 .loss/FRt2/detection_loss‚1.230384600826527 1loss/SRqqt1/detection_loss‚1.0089381740350898 %loss/total_loss‚8.729113833197097 $ _timestamp‚1.784280428537639e+09 _step‚737 0loss/FRt2/assignment_loss‚0.8646994134711865 /loss/FRt1/detection_loss‚0.5009770781319745 epoch‚22 1loss/SRqqt2/detection_loss‚0.5608376121557859 0loss/FBt1/detection_loss‚0.49719629553342315 /loss/FBt2/detection_loss‚1.3173363695830285 3loss/SRqqt2/assignment_loss‚0.49163905458480783 1loss/FBt1/assignment_loss‚0.04550737126876871 2loss/FBt2/assignment_loss‚0.021731880524720554Ç*²À ܇ 3loss/SRqqt2/assignment_loss‚0.49163905458480783 1loss/FBt1/assignment_loss‚0.04550737126876871 2loss/FBt2/assignment_loss‚0.021731880524720554 trainer/global_step‚8949 _runtime‚7620.663016334 /loss/FRt1/assignment_loss‚1.673819718727899 2loss/SRqqt1/assignment_loss‚0.5160462643538873 .loss/FRt2/detection_loss‚1.230384600826527 1loss/SRqqt1/detection_loss‚1.0089381740350898 %loss/total_loss‚8.729113833197097 $ _timestamp‚1.784280428537639e+09 _step‚737 0loss/FRt2/assignment_loss‚0.8646994134711865 /loss/FRt1/detection_loss‚0.5009770781319745 epoch‚22 1loss/SRqqt2/detection_loss‚0.5608376121557859 0loss/FBt1/detection_loss‚0.49719629553342315 /loss/FBt2/detection_loss‚1.3173363695830285áÕÌßè“Ý‚(:ˆ úêçÒ׫֤ gpu.0.gpu‚78.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚205.274' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.0548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n"ÈÓÞ‚(:÷ úêçÒÞÓߦ disk.nvme0n1.in‚ 371.04296875 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30406.86962890625 disk.sda2.out‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.21434116363525( proc.memory.availableMB‚ 456.40234375 disk.dm-0.in‚ 371.04296875 disk.drbd2165.out‚0 cpu‚26.044956720204592& disk.nvme0n1.out‚30406.86962890625 disk.sda.in‚0 disk.sda3.out‚ 280.0390625 disk.dm-1.in‚0 disk.dm-1.out‚ 280.0390625* proc.memory.percent‚17.149853706359863 network.sent‚ 1.4206397e+07# disk.dm-0.out‚30406.86962890625" disk.nvme0c0n1.in‚ 371.04296875 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 2.018291e+06 disk.sda.out‚ 280.0390625 proc.cpu.threads‚17) disk./.usagePercent‚36.61867716688054$ disk./.usageGB‚545.6447525024414 disk.sda2.in‚0 disk.rbd1.out‚102.9189453125# proc.memory.rssMB‚ 2809.83203125;A2“ß‚(:ˆ ‰ëçÒ뮬§ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ò°^à‚(:ø ‰ëçÒÊà˪* disk./.usagePercent‚36.618610092906565 disk.sda3.out‚ 280.2734375 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 371.04296875 disk.sda2.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 371.04296875# disk.dm-0.out‚30412.35302734375( proc.memory.availableMB‚ 456.45703125& disk.nvme0n1.out‚30412.35302734375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 280.2734375 disk.rbd3.in‚0 network.recv‚ 2.023611e+06( disk.nvme0c0n1.out‚30412.35302734375 disk.rbd1.out‚102.9189453125 disk.dm-0.in‚ 371.04296875 cpu‚26.045491438459024$ disk./.usageGB‚545.6437530517578 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 280.2734375 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2809.83203125* proc.memory.percent‚17.149853706359863 network.sent‚ 1.4271516e+07 disk.sda2.out‚0 disk.rbd3.out‚0$ memory_percent‚97.21400737762451 proc.cpu.threads‚17!SÚ…2á- _wandbruntime‚7660  _runtime‚7660çþ—ªÇâÁ _step‚738 & lr-AdamW/pg1‚0.0002815485925223544 & lr-AdamW/pg2‚0.0002815485925223544 trainer/global_step‚8999 % _timestamp‚1.7842804685087173e+09 _runtime‚7660.634361365L-NÌÌãÆ & lr-AdamW/pg1‚0.0002815485925223544 & lr-AdamW/pg2‚0.0002815485925223544 trainer/global_step‚8999 % _timestamp‚1.7842804685087173e+09 _runtime‚7660.634361365 _step‚738â½ÊqZ2ä- _wandbruntime‚7660  _runtime‚7660m1Ffô'åîO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301332871825408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9501392239145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19385076 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚310954972184576 % _timestamp‚1.7842804685098608e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚739 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35151667 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1370123296573993 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1377171468804608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83355281 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9622015976448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19385346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83355281 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9622100359168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚8999 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1377171827183104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35151394 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1367655556939776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9501732789035 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1377171827183104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9515911864832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154691816 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1377171468804608 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1367655573979136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9516253203968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9515911864832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888[ïßMæóO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9515911864832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301332871825408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9501392239145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19385076 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚310954972184576 % _timestamp‚1.7842804685098608e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚739 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35151667 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1370123296573993 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1377171468804608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83355281 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9622015976448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19385346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83355281 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9622100359168 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚8999 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1377171827183104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35151394 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1367655556939776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9501732789035 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1377171827183104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9515911864832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154691816 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1377171468804608 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1367655573979136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9516253203968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616㶈2ç- _wandbruntime‚7661  _runtime‚7661@ f þ&èøM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9502403292867 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9502743842869 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83364347 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83364347 trainer/global_step‚8999 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9516924491776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301366281971712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71354144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1367802815008768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154718493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19387184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310989424628736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9623037572096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9516924491776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301366300583936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15768063 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154718493 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154708500 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35155247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154708500 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1377319722461184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7842804691214502e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83354356 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 _step‚740 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1377320080846848 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71354146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1360768432046080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1377320080846848 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71354146 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1360768432046080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83364347 trainer/global_step‚8999 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9516924491776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301366281971712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71354144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1367802815008768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154718493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19387184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310989424628736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9623037572096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9516924491776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301366300583936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15768063 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154718493 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154708500 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35155247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154708500 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1377319722461184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7842804691214502e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83354356 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 _step‚740 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ä›BÄ|2ê- _wandbruntime‚7661  _runtime‚7661º±º6ë /loss/FBt1/detection_loss‚0.4161150765998276 0loss/FRt2/assignment_loss‚0.9062206722134328 3loss/SRqqt1/assignment_loss‚0.47817146510255626 2loss/SRqqt2/assignment_loss‚0.4853768011509491 2loss/FBt2/assignment_loss‚0.004106399271625112 .loss/FRt1/detection_loss‚0.529160047957865 $ _timestamp‚1.784280469125143e+09 _step‚741 1loss/SRqqt1/detection_loss‚1.0471485561554021 1loss/SRqqt2/detection_loss‚0.6957866610954199 %loss/total_loss‚8.632073004438325 epoch‚22 _runtime‚7661.250477825 /loss/FRt1/assignment_loss‚1.650288336406088 2loss/FBt1/assignment_loss‚0.006223186121087434 .loss/FRt2/detection_loss‚1.264960664559091 trainer/global_step‚8999 /loss/FBt2/detection_loss‚1.1485151378049825—³aä ì† /loss/FRt1/assignment_loss‚1.650288336406088 2loss/FBt1/assignment_loss‚0.006223186121087434 .loss/FRt2/detection_loss‚1.264960664559091 trainer/global_step‚8999 /loss/FBt2/detection_loss‚1.1485151378049825 /loss/FBt1/detection_loss‚0.4161150765998276 0loss/FRt2/assignment_loss‚0.9062206722134328 3loss/SRqqt1/assignment_loss‚0.47817146510255626 2loss/SRqqt2/assignment_loss‚0.4853768011509491 2loss/FBt2/assignment_loss‚0.004106399271625112 .loss/FRt1/detection_loss‚0.529160047957865 $ _timestamp‚1.784280469125143e+09 _step‚741 1loss/SRqqt1/detection_loss‚1.0471485561554021 1loss/SRqqt2/detection_loss‚0.6957866610954199 %loss/total_loss‚8.632073004438325 epoch‚22 _runtime‚7661.250477825å™É7@“í‚(:ˆ ˜ëçÒ¯¡µ¥ gpu.0.gpu‚59.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä)•î‚(:ù ˜ëçÒÍ⧠disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd1.out‚102.9814453125& disk.nvme0n1.out‚30419.33544921875 network.sent‚ 1.4287479e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 280.40234375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 371.87109375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 371.87109375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.21388816833496' proc.memory.availableMB‚ 456.4765625# proc.memory.rssMB‚ 2809.83203125( disk.nvme0c0n1.out‚30419.33544921875 cpu‚26.04590936874574* disk./.usagePercent‚36.618653358179856 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 371.87109375 disk.sda.in‚0 proc.cpu.threads‚17 network.recv‚ 2.026731e+06$ disk./.usageGB‚545.6443977355957 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda.out‚ 280.40234375 disk.sda3.out‚ 280.40234375# disk.dm-0.out‚30419.33544921875 disk.drbd2165.out‚0* proc.memory.percent‚17.149853706359863HúOJï‚(:’ §ëçÒû•š¥ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚272.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.571999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡oÂ.ùð‚(:î §ëçÒ¨ö̧) disk./.usagePercent‚36.61863006149424 disk.dm-0.in‚ 372.5859375# proc.memory.rssMB‚ 2809.83203125 disk.rbd3.out‚0$ disk./.usageGB‚545.6440505981445 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 372.5859375 disk.sda.in‚0 disk.rbd1.out‚102.9814453125* proc.memory.percent‚17.149853706359863 network.recv‚ 2.032392e+06! disk.dm-0.out‚30428.611328125$ disk.nvme0n1.out‚30428.611328125 disk.rbd0.out‚0& disk.nvme0c0n1.out‚30428.611328125 disk.dm-1.in‚0% proc.memory.availableMB‚ 456.53125 disk.drbd2165.out‚0 disk.sda3.out‚ 280.66015625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd0.in‚0 disk.sda.out‚ 280.66015625 disk.sda3.in‚0$ memory_percent‚97.21355438232422 disk.nvme0n1.in‚ 372.5859375 disk.sda2.in‚0 network.sent‚ 1.4339865e+07 disk.dm-1.out‚ 280.66015625 disk.rbd3.in‚0 cpu‚26.046598056225296ÜCRžñ‚(:“ ¶ëçÒé¶þ¤ gpu.0.gpu‚70.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.510799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àíþò‚(:ó ¶ëçÒ¶¾¦§ disk.dm-1.out‚ 280.9140625 disk.rbd3.in‚0 network.sent‚ 1.4344872e+07 disk.sda.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 455.91796875( disk.nvme0c0n1.out‚30435.20068359375 disk.rbd1.out‚103.0439453125 network.recv‚ 2.035601e+06 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.dm-0.in‚ 373.4140625# proc.memory.rssMB‚ 2809.83203125) disk./.usagePercent‚36.61870046356616& disk.nvme0n1.out‚30435.20068359375 disk.sda.out‚ 280.9140625 disk.nvme0n1.in‚ 373.4140625 disk.drbd2165.in‚0# disk.dm-0.out‚30435.20068359375* proc.memory.percent‚17.149853706359863 cpu‚26.04712351504135$ disk./.usageGB‚545.6450996398926 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 373.4140625 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚97.21729755401611 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 280.9140625\pI|2ó- _wandbruntime‚7701  _runtime‚7701 C%µÉôà ' lr-AdamW/pg2‚0.00027968810161085617 trainer/global_step‚9049 % _timestamp‚1.7842805090517707e+09 _runtime‚7701.177365909 _step‚742 ' lr-AdamW/pg1‚0.00027968810161085617aÎõÈ ' lr-AdamW/pg1‚0.00027968810161085617 ' lr-AdamW/pg2‚0.00027968810161085617 trainer/global_step‚9049 % _timestamp‚1.7842805090517707e+09 _runtime‚7701.177365909 _step‚742æÀéâ£2ö- _wandbruntime‚7701  _runtime‚7701s –íÚ÷íO ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1375017608421376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35343035 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1384584215788032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚9049 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9673250812416 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1377499804243751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1375017625460736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚743 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71728114 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71728114 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9566607366656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9552349275943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303004291139584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1367947437928448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842805090527787e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1375017625460736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1384584574166528 eQDeviceStatsMonitor.on_train_batch_start/i‰Oñnactive_split_bytes.large_pool.allocated‚303004309751808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83799756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9566948705792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35343300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19489467 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15853568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1367947454967808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9673331000832 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71728112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19489729 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83809733 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9566607366656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15853571 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312677541952000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9552008726053 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9566948705792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155537847 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚7701.182369896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155527868 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088<lø'øòO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9673331000832 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71728112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19489729 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83809733 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9566607366656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15853571 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312677541952000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9552008726053 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9566948705792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155537847 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚7701.182369896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155527868 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1375017608421376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35343035 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1384584215788032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚9049 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9673250812416 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1377499804243751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1375017625460736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚743 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71728114 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71728114 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9566607366656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9552349275943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303004291139584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1367947437928448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842805090527787e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1375017625460736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1384584574166528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303004309751808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83799756 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9566948705792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35343300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19489467 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15853568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1367947454967808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888ç%Eo2ù- _wandbruntime‚7701  _runtime‚7701BÔ}íþ&úøM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35347110 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1377647325969457 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19491535 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83808808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9567961339904 % _timestamp‚1.7842805096645586e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9567619993600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9567619993600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83808808 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1375164849451008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9674353105920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1384732827830272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚7701.790409734 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155554545 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1377646968380095 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚9049 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15855316 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1375164866490368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71735744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312712091616256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303037719898112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1375164866490368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1375164849451008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303037738510336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1368093965639680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9567961339904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155544552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71735746 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71735746 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155554545 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35346848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19491794 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1384732827830272 `ëðœûýM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155554545 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1377646968380095 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚9049 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15855316 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1375164866490368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71735744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312712091616256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303037719898112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1375164866490368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1375164849451008 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303037738510336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1368093965639680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9567961339904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155544552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71735746 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71735746 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155554545 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35346848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19491794 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1384732827830272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35347110 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1377647325969457 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19491535 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83808808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9567961339904 % _timestamp‚1.7842805096645586e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9567619993600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9567619993600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83808808 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1375164849451008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9674353105920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1384732827830272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚7701.790409734 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Bþó'‰íO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1375272971522048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1382379659902976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1391996962771456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314400097658368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304675729065984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1382379676942336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84264185 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1391997321149952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84254208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1391996962771456 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9617644207616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72109714 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156373899 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1384875596735009 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84254208 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1391997321149952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9617302868480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1375272988561408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1382379659902976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9602625212961 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1382379676942336 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1384875954324259 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 % _timestamp‚1.7842805493187933e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9724368592384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156363920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35533498 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19592677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9724463460864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304675747678208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9602965762851 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9617302868480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84264185 _runtime‚ 7741.44459574»ˆ‘–ø'ŠòO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15940818 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚9099 _step‚747 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35533227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15940821 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9724368592384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156363920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35533498 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19592677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9724463460864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304675747678208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9602965762851 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9617302868480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84264185 _runtime‚ 7741.44459574 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1375272971522048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1382379659902976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1391996962771456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314400097658368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304675729065984 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1382379676942336 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84264185 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1391997321149952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84254208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1391996962771456 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9617644207616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72109714 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156373899 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1384875596735009 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84254208 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1391997321149952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9617302868480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1375272988561408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1382379659902976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9602625212961 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1382379676942336 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1384875954324259 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 % _timestamp‚1.7842805493187933e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceSt í6  atsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156390597 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72117346 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1382526900932608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9725385330176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1382526917971968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35537001 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15942566 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784280549931539e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19594438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156390597 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9603636266683 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9618656841728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72117344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84273251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72117344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72117346 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314434543154688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1375419482193920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9618315495424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1392145574813696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84263260 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35537269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1382526900932608 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1392145574813696 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156390597 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72117346 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1382526900932608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9725385330176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1382526917971968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35537001 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15942566 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784280549931539e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19594438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156390597 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9603636266683 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9618656841728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72117344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84273251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72117344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72117346 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314434543154688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1375419482193920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9618315495424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ì„u?y2Ž- _wandbruntime‚7742  _runtime‚7742†4ƒö † 1loss/SRqqt1/detection_loss‚0.9098106662904393 /loss/FBt2/detection_loss‚0.8469543419778347 2loss/FBt1/assignment_loss‚0.009174274680273715 trainer/global_step‚9099 0loss/FBt1/detection_loss‚0.44614449163077413 %loss/total_loss‚7.991927054787727 epoch‚22 % _timestamp‚1.7842805499345996e+09 0loss/FRt1/assignment_loss‚1.5746013441426783 2loss/SRqqt2/assignment_loss‚0.5233385458750467 3loss/FBt2/assignment_loss‚0.0005890403903790391 /loss/FRt1/detection_loss‚0.5072050057942258 /loss/FRt2/detection_loss‚1.3098291255499788 1loss/SRqqt2/detection_loss‚0.6181276517631817 _runtime‚7742.059939733 _step‚749 0loss/FRt2/assignment_loss‚0.8459520154786342 2loss/SRqqt1/assignment_loss‚0.4002005512142803€²¬‹ 1loss/SRqqt2/detection_loss‚0.6181276517631817 _runtime‚7742.059939733 _step‚749 0loss/FRt2/assignment_loss‚0.8459520154786342 2loss/SRqqt1/assignment_loss‚0.4002005512142803 1loss/SRqqt1/detection_loss‚0.9098106662904393 /loss/FBt2/detection_loss‚0.8469543419778347 2loss/FBt1/assignment_loss‚0.009174274680273715 trainer/global_step‚9099 0loss/FBt1/detection_loss‚0.44614449163077413 %loss/total_loss‚7.991927054787727 epoch‚22 % _timestamp‚1.7842805499345996e+09 0loss/FRt1/assignment_loss‚1.5746013441426783 2loss/SRqqt2/assignment_loss‚0.5233385458750467 3loss/FBt2/assignment_loss‚0.0005890403903790391 /loss/FRt1/detection_loss‚0.5072050057942258 /loss/FRt2/detection_loss‚1.3098291255499788íµÇk“‘‚(:ˆ òëçÒß”­¥ gpu.0.gpu‚90.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_Þ’ô’‚(:ö òëçÒÆ‹Ú§& disk.nvme0n1.out‚30463.78271484375 disk.drbd2165.in‚0$ memory_percent‚97.21639156341553* proc.memory.percent‚17.149853706359863( disk.nvme0c0n1.out‚30463.78271484375 network.recv‚ 2.050583e+06 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 375.671875 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2809.83203125 disk.sda3.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 375.671875( proc.memory.availableMB‚ 456.06640625 disk.dm-1.out‚ 281.72265625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0 network.sent‚ 1.4420084e+07 disk.rbd1.in‚0.0625# disk.dm-0.out‚30463.78271484375 cpu‚26.049194018418223$ disk./.usageGB‚545.6415939331055 disk.sda3.out‚ 281.72265625! disk.rbd1.out‚103.25537109375 disk.nvme0c0n1.in‚ 375.671875 disk.sda.out‚ 281.72265625 proc.cpu.threads‚17* disk./.usagePercent‚36.618465192642184~Ç5š“‚(:’ ìçÒÑÒϤ gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q­»â(:ã ìçÒòõÙ¦ disk.nvme0n1.in‚376.5 disk.sda1.out‚0 disk.nvme0c0n1.in‚376.5 disk.dm-1.out‚ 281.96484375$ memory_percent‚97.21598625183105 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 456.1328125* proc.memory.percent‚17.149853706359863$ disk./.usageGB‚545.6459121704102 disk.rbd3.in‚0" disk.dm-0.out‚30473.1279296875 disk.drbd2165.out‚0 disk.sda.in‚0 cpu‚26.049746296700864 network.recv‚ 2.055682e+06* disk./.usagePercent‚36.618754993170946 disk.dm-0.in‚376.5' disk.nvme0c0n1.out‚30473.1279296875 disk.dm-1.in‚0# proc.memory.rssMB‚ 2809.83203125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda.out‚ 281.96484375 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚103.25537109375 network.sent‚ 1.4485228e+07% disk.nvme0n1.out‚30473.1279296875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 281.96484375"íà§2•- _wandbruntime‚7781  _runtime‚7781¿ÆØ“É–Ã _step‚750 ' lr-AdamW/pg1‚0.00027595888582437265 ' lr-AdamW/pg2‚0.00027595888582437265 trainer/global_step‚9149 % _timestamp‚1.7842805898347383e+09 _runtime‚7781.960413148=sÎ—È ' lr-AdamW/pg1‚0.00027595888582437265 ' lr-AdamW/pg2‚0.00027595888582437265 trainer/global_step‚9149 % _timestamp‚1.7842805898347383e+09 _runtime‚7781.960413148 _step‚750î ¬Ù2˜- _wandbruntime‚7781  _runtime‚7781/‰‰,%™îO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9668339709440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72491312 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306347166992384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84708660 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19695352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1399409709754880 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1389741711384576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316122629234176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35723172 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1392252104404767 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72491314 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚7781.961625026 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9668339709440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16028068 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1399409709754880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84708660 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316122740617728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1382598505115648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9775462241792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9653582249759 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1399410068133376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35723423 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16028071 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚306347185604608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9653241699869 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72491312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84718637™†=cù'šóO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1389741728423936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1389741711384576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19695104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9667998370304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72491314 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1382598522155008 trainer/global_step‚9149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157199972 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306347166992384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84708660 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19695352 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1399409709754880 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1389741711384576 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316122629234176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35723172 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1392252104404767 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72491314 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚7781.961625026 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9668339709440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16028068 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1399409709754880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84708660 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316122740617728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1382598505115648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9775462241792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9653582249759 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1399410068133376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35723423 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16028071 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚306347185604608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9653241699869 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72491312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84718637 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9668339709440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72491312 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157226649 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚306380614363136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84727703 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1389888969453568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1392399626130473 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚7782.573991033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1399558321797120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84717712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35726972 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9654252753591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399557963411456 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1382745032826880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157226649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1389888969453568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19697423 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72498946 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306380595750912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72498944 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1399558321797120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9669352343552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19697159 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9654593303593 % _timestamp‚1.7842805904481795e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84727703 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9776575709184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9776478750720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1389888952414208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157216656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35727239 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16029813 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚9149 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1389888952414208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316157190072320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1382745015787520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316157074501632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316157074501632 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157226649 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚306380614363136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84727703 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1389888969453568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1392399626130473 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚7782.573991033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1399558321797120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84717712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35726972 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9654252753591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399557963411456 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1382745032826880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157226649 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1389888969453568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19697423 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72498946 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306380595750912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72498944 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1399558321797120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9669352343552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19697159 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9654593303593 % _timestamp‚1.7842805904481795e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84727703 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9776575709184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9776478750720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1389888952414208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157216656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35727239 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16029813 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚9149 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1389888952414208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316157190072320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1382745015787520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 Fz2ž- _wandbruntime‚7782  _runtime‚7782vþ  Ÿ… /loss/FRt1/detection_loss‚0.5512971825899095 % _timestamp‚1.7842805904501514e+09 _runtime‚7782.575476387 1loss/SRqqt1/detection_loss‚0.9323852998648936 1loss/SRqqt2/detection_loss‚0.6760732897619209 /loss/FBt2/detection_loss‚0.7668128520516413 $loss/total_loss‚8.01800043803653 3loss/SRqqt2/assignment_loss‚0.46338768442840694 3loss/FBt2/assignment_loss‚0.0017132294999948933 .loss/FRt2/detection_loss‚1.194586499478468 0loss/FBt1/detection_loss‚0.38745247983466835 epoch‚22 trainer/global_step‚9149 0loss/FRt1/assignment_loss‚1.6323483954705662 0loss/FRt2/assignment_loss‚0.8444266753736047 2loss/SRqqt1/assignment_loss‚0.5573748264657724 2loss/FBt1/assignment_loss‚0.010142023216684152 _step‚753#gê Š 3loss/SRqqt2/assignment_loss‚0.46338768442840694 3loss/FBt2/assignment_loss‚0.0017132294999948933 .loss/FRt2/detection_loss‚1.194586499478468 0loss/FBt1/detection_loss‚0.38745247983466835 epoch‚22 trainer/global_step‚9149 0loss/FRt1/assignment_loss‚1.6323483954705662 0loss/FRt2/assignment_loss‚0.8444266753736047 2loss/SRqqt1/assignment_loss‚0.5573748264657724 2loss/FBt1/assignment_loss‚0.010142023216684152 _step‚753 /loss/FRt1/detection_loss‚0.5512971825899095 % _timestamp‚1.7842805904501514e+09 _runtime‚7782.575476387 1loss/SRqqt1/detection_loss‚0.9323852998648936 1loss/SRqqt2/detection_loss‚0.6760732897619209 /loss/FBt2/detection_loss‚0.7668128520516413 $loss/total_loss‚8.01800043803653ñ‰2Dÿ€¡‚(:’ ìçÒ••¿¥ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAÍ£llocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žjca袂(:Ý ìçÒ†Ëç* proc.memory.percent‚17.149853706359863 disk.drbd2165.out‚0 network.recv‚ 2.058896e+06! disk.rbd1.out‚103.31787109375 disk.nvme0n1.in‚376.5 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0$ memory_percent‚97.21593856811523 disk.rbd1.in‚0.0625 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 282.109375 disk.rbd3.in‚0# disk.dm-0.out‚30475.95263671875& disk.nvme0n1.out‚30475.95263671875 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚30475.95263671875 disk.sda3.in‚0) disk./.usagePercent‚36.61875729723876$ disk./.usageGB‚545.6459465026855 disk.dm-0.in‚376.5 disk.nvme0c0n1.in‚376.5 cpu‚26.050251822999222 proc.cpu.threads‚17 network.sent‚ 1.448815e+07 disk.rbd3.out‚0 disk.sda.out‚ 282.109375 disk.sda2.out‚0# proc.memory.rssMB‚ 2809.83203125 disk.sda3.out‚ 282.109375& proc.memory.availableMB‚ 456.140625Œ´ž£‚(:“ ŸìçÒ‡“ͤ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚182.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.510799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜžÇþ¤‚(:ó ŸìçÒÖ†ý¦ network.recv‚ 2.064414e+06' disk.nvme0c0n1.out‚30485.9072265625 disk.sda2.in‚0 disk.dm-1.out‚ 283.3046875" disk.dm-0.out‚30485.9072265625 cpu‚26.050892527090458 disk.nvme0n1.in‚ 377.21484375% disk.nvme0n1.out‚30485.9072265625* proc.memory.percent‚17.149853706359863) disk./.usagePercent‚36.61917356548943 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 377.21484375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 283.3046875 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 283.3046875 disk.rbd1.in‚0.0625 network.sent‚ 1.4553354e+07 disk.sda2.out‚0# proc.memory.rssMB‚ 2809.83203125$ disk./.usageGB‚545.6521492004395" disk.nvme0c0n1.in‚ 377.21484375 disk.dm-1.in‚0! disk.rbd1.out‚103.31787109375 disk.rbd3.in‚0$ memory_percent‚97.21689224243164& proc.memory.availableMB‚ 455.984375 proc.cpu.threads‚17P4ºXž¥‚(:“ ®ìçÒÈ´§ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚199.034' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.806799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_³*°ú¦‚(:ï ®ìçÒ¨ÉÞ«( proc.memory.availableMB‚ 455.67578125% disk.nvme0n1.out‚31951.3740234375 disk.sda1.out‚0 disk.sda.out‚ 284.7734375 disk.sda2.out‚0! disk.rbd1.out‚103.38037109375) disk./.usagePercent‚36.76009956890789 disk.rbd3.in‚0 disk.dm-0.in‚ 378.046875 network.recv‚ 2.067641e+06 disk.drbd2165.in‚0 disk.rbd0.in‚0$ memory_percent‚97.21877574920654* proc.memory.percent‚17.149853706359863 disk.sda.in‚0 cpu‚26.051366583669367 network.sent‚ 1.4558365e+07 disk.sda3.out‚ 284.7734375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.nvme0n1.in‚ 378.046875 disk.sda3.in‚0 disk.dm-1.in‚0$ disk./.usageGB‚547.7520484924316 disk.nvme0c0n1.in‚ 378.046875' disk.nvme0c0n1.out‚31951.3740234375 disk.dm-1.out‚ 284.7734375 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2809.83203125" disk.dm-0.out‚31951.3740234375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0ßOØ2§- _wandbruntime‚7823  _runtime‚78230¼ cƨÀ $ _timestamp‚1.784280631335504e+09 _runtime‚7823.461073258 _step‚754 & lr-AdamW/pg1‚0.0002740904004699486 & lr-AdamW/pg2‚0.0002740904004699486 trainer/global_step‚9199SE…fË©Å & lr-AdamW/pg1‚0.0002740904004699486 & lr-AdamW/pg2‚0.0002740904004699486 trainer/global_step‚9199 $ _timestamp‚1.784280631335504e+09 _runtime‚7823.461073258 _step‚754ò]‹¦2ª- _wandbruntime‚7823  _runtime‚7823‡’ô'«îO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9826763100672 % _timestamp‚1.7842806313366094e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85173089 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85163112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19797699 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚755 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158036024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158046003 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1406822815116800 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317845386631680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1406822815116800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35913277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16115321 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308018623531008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1397103779905536 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1397103762866176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85173089 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1406822456738304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308018604918784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9704198736667 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9826672426496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1389924055748608 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9718693872128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚7823.462379755 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72872914 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72872912 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1406822456738304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72872912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72872914 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1399627896896025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16115318 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1389924038709248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9719035211264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9719035211264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9718693872128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317845277345280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1406822815116800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35913277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16115321 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308018623531008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1397103779905536 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1397103762866176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85173089 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1406822456738304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308018604918784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9704198736667 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9826672426496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1389924055748608 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9718693872128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚7823.462379755 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72872914 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72872912 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1406822456738304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72872912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72872914 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1399627896896025 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16115318 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1389924038709248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9719035211264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9719035211264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9718693872128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317845277345280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16117066 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85182155 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1397251020935168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9705209790501 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158062701 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9720047845376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19799726 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1390070549381120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1397251020935168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85172164 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35917071 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1390070566420480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9827783989248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚9199 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9719706499072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35916789 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9720047845376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1399775418621619 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16117063 E?DeviceStats"èÜr­ Monitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72880546 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1397251003895808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19800005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9704869240499 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317879836278784 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842806319692805e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72880544ZZÊ '¯ýM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85182155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317879722805248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308052052289536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16117066 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85182155 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1397251020935168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9705209790501 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158062701 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9720047845376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19799726 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1390070549381120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1397251020935168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85172164 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35917071 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1390070566420480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9827783989248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚9199 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9719706499072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35916789 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9720047845376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1399775418621619 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16117063 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72880546 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1397251003895808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19800005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9704869240499 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317879836278784 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842806319692805e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72880544 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 % _timestamp‚1.7842806500851092e+09 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996282696723938 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7028476595878601 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8220334053039551 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25470221042633057 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6157894730567932 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2753177285194397 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_2_of_3‚0.5783528685569763 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083134651184 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2866179943084717 7validation_average_jet_accuracy‚0.38837388157844543 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2305581420660019 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23936350643634796 /particle/accuracy_4_of_4‚0.9449505805969238 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8056312799453735 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27416929602622986 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6600770354270935 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15256474912166595 particle/accuracy_3_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28569769859313965 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16873151063919067 *jet/accuracy_1_of_3‚0.9316525459289551 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9099478721618652 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_2‚0.1597372591495514 +jet/accuracy_3_of_4‚0.08897139132022858 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2861587107181549 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚8.009472035161645 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5049441456794739 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12166666984558105 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 )jet/accuracy_2_of_4‚0.781094491481781 +particle/sensitivity‚0.6172108054161072 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458730697632 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6993046998977661 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31484878063201904 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8647398948669434 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8526846766471863 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2659949064254761 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5135048627853394 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6552702188491821 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7245668768882751 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9892179369926453 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.65241539478302 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2753177285194397 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7927060127258301 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3890326917171478 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030387070029973984 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6558434367179871 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03912629559636116 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8523942232131958 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23420247435569763 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7396982908248901 +jet/accuracy_3_of_3‚0.07350017875432968 _step‚758 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24627594649791718 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5092185139656067 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_3_of_3‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029377855360507965 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7856107354164124 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7275810241699219 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7425405383110046 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6628249287605286 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30826860666275024 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 jet/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7946522831916809 _runtime‚7842.211293881 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2182491421699524 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987881183624268 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.208228200674057 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6002922058105469 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05773137882351875 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6558434367179871 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7253607511520386 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_4‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7796691656112671 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9758117198944092 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2150670737028122 *jet/accuracy_1_of_2‚0.8037432432174683 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5851286053657532 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6365103125572205 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7269781827926636 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2287653535604477 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7374826669692993 jet/accuracy_3_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.726488471031189 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959500432014465 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6828942894935608 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23493225872516632 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6422598958015442 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9252536296844482 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.811262309551239 'particle/accuracy‚0.847082793712616 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23653987050056458 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.035853173583745956 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3481110632419586 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6834592223167419 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2484375685453415 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993018627166748 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 epoch‚22 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7627282738685608 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6671778559684753 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7008234858512878 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050678689032793045 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03188255801796913 ,jet/accuracy_4_of_4‚0.011098379269242287 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6469887495040894 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06769022345542908 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030397702008485794 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6"Purity/1FRt*SRqqt2FBt/event_purity‚0.7219198346138 'particle/f_score‚0.6341648697853088 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5956974029541016 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9052057862281799 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.652188777923584 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5109429359436035 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3880400061607361 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.712287962436676 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7196084260940552 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4722222089767456 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3488834798336029 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0ì8··{ :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03912629559636116 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8523942232131958 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23420247435569763 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7396982908248901 +jet/accuracy_3_of_3‚0.07350017875432968 _step‚758 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24627594649791718 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5092185139656067 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_3_of_3‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029377855360507965 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7856107354164124 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7275810241699219 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7425405383110046 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6628249287605286 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30826860666275024 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/ù·´…8*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 jet/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7946522831916809 _runtime‚7842.211293881 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2182491421699524 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987881183624268 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.208228200674057 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6002922058105469 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05773137882351875 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6558434367179871 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7253607511520386 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_4‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7796691656112671 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9758117198944092 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2150670737028122 *jet/accuracy_1_of_2‚0.8037432432174683 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5851286053657532 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6365103125572205 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7269781827926636 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2287653535604477 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7374826669692993 jet/accuracy_3_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.726488471031189 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959500432014465 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6828942894935608 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23493225872516632 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6422598958015442 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9252536296844482 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.811262309551239 'particle/accuracy‚0.847082793712616 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23653987050056458 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.035853173583745956 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3481110632419586 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6834592223167419 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2484375685453415 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993018627166748 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 epoch‚22 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7627282738685608 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6671778559684753 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7008234858512878 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050678689032793045 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03188255801796913 ,jet/accuracy_4_of_4‚0.011098379269242287 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6469887495040894 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06769022345542908 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030397702008485794 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6"Purity/1FRt*SRqqt2FBt/event_purity‚0.7219198346138 'particle/f_score‚0.6341648697853088 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5956974029541016 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9052057862281799 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.652188777923584 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5109429359436035 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3880400061607361 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.712287962436676 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7196084260940552 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4722222089767456 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3488834798336029 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9893644452095032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6682751774787903 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 trainer/global_step‚9199 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7865274548530579 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6857492923736572 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5286051630973816 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02540370635688305 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5178511738777161 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02540370635688305 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33126598596572876 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.030724378302693367 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8523942232131958 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18104569613933563 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7565541863441467 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2589457929134369 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1648438274860382 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7855560779571533 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.273040771484375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4215939939022064 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 % _timestamp‚1.7842806500851092e+09 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996282696723938 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7028476595878601 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8220334053039551 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25470221042633057 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6157894730567932 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2753177285194397 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_2_of_3‚0.5783528685569763 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083134651184 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2866179943084717 7validation_average_jet_accuracy‚0.38837388157844543 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2305581420660019 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23936350643634796 /particle/accuracy_4_of_4‚0.9449505805969238 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8056312799453735 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27416929602622986 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6600770354270935 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15256474912166595 particle/accuracy_3_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28569769859313965 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16873151063919067 *jet/accuracy_1_of_3‚0.9316525459289551 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9099478721618652 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_2‚0.1597372591495514 +jet/accuracy_3_of_4‚0.08897139132022858 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2861587107181549 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚8.009472035161645 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5049441456794739 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12166666984558105 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 )jet/accuracy_2_of_4‚0.781094491481781 +particle/sensitivity‚0.6172108054161072 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458730697632 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6993046998977661 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31484878063201904 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8647398948669434 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8526846766471863 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2659949064254761 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5135048627853394 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6552702188491821 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7245668768882751 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9892179369926453 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.65241539478302 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2753177285194397 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7927060127258301 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3890326917171478 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030387070029973984 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6558434367179871 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067ö“˜J”‘¸‚(:† ÌìçÒЇä£ gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þixòÿ¹‚(:ô Ììçқ蟧 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 288.3828125! disk.rbd1.out‚103.44287109375 disk.rbd3.in‚0' proc.memory.availableMB‚ 455.1640625 network.recv‚ 2.076416e+06) disk./.usagePercent‚36.64091986947794 disk.sda3.in‚0 disk.dm-0.in‚ 379.60546875) proc.memory.percent‚17.15383529663086 network.sent‚ 1.4637179e+07 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 379.60546875 disk.sda.out‚ 288.3828125 cpu‚26.060137104908886 disk.dm-1.in‚0 disk.dm-1.out‚ 288.3828125! proc.memory.rssMB‚ 2810.484375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚32928.76708984375$ memory_percent‚97.22189903259277$ disk./.usageGB‚545.9761848449707# disk.dm-0.out‚32928.76708984375 disk.nvme0n1.in‚ 379.60546875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚32928.76708984375 disk.sda.in‚0 disk.sda2.in‚0\™KKžº‚(:“ ÛìçÒÝ㤠gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J%À¥þ»‚(:ó ÛìçÒÒ«ø¥ disk.sda.out‚ 288.89453125 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚97.22151756286621 network.recv‚ 2.081562e+06" disk.nvme0c0n1.in‚ 379.60546875 disk.sda2.out‚0 disk.nvme0n1.in‚ 379.60546875% disk.nvme0n1.out‚32964.7353515625' disk.nvme0c0n1.out‚32964.7353515625 disk.sda2.in‚0! proc.memory.rssMB‚ 2810.484375 proc.cpu.threads‚17 network.sent‚ 1.4683823e+07 disk.rbd3.in‚0) disk./.usagePercent‚36.80572804774307" disk.dm-0.out‚32964.7353515625 cpu‚26.06056956169783 disk.sda1.in‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 455.2265625) proc.memory.percent‚17.15383529663086 disk.dm-1.in‚0 disk.dm-1.out‚ 288.89453125! disk.rbd1.out‚103.44287109375 disk.rbd3.out‚0 disk.dm-0.in‚ 379.60546875 disk.sda3.out‚ 288.89453125$ disk./.usageGB‚548.4319458007812 disk.rbd0.in‚0 disk.rbd0.out‚0òz`‘¼‚(:† êìçÒð컢 gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ûr½‚(:÷ êìçÒ†ïÆ¥ disk.dm-1.in‚0! disk.rbd1.out‚103.50537109375$ memory_percent‚97.22151756286621) proc.memory.percent‚17.15383529663086 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 290.83203125 disk.sda1.in‚0 cpu‚26.061309776422824 disk.sda2.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚33178.01904296875 disk.nvme0n1.in‚ 380.32421875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 380.32421875( disk.nvme0c0n1.out‚33178.01904296875 disk.sda1.out‚0! proc.memory.rssMB‚ 2810.484375 disk.rbd1.in‚0.0625) disk./.usagePercent‚36.83459980943209$ disk./.usageGB‚548.8621559143066 disk.dm-1.out‚ 290.83984375 disk.dm-0.in‚ 380.32421875& disk.nvme0n1.out‚33178.01904296875' proc.memory.availableMB‚ 455.2265625 network.sent‚ 1.4688878e+07 disk.sda2.out‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 290.79296875 proc.cpu.threads‚17 network.recv‚ 2.084876e+06 disk.sda.in‚0þ öG2¾- _wandbruntime‚7882  _runtime‚7882ù–YžÇ¿Á _step‚759 & lr-AdamW/pg1‚0.0002722194900993462 & lr-AdamW/pg2‚0.0002722194900993462 trainer/global_step‚9249 % _timestamp‚1.7842806901227295e+09 _runtime‚7882.248422017 åêÖÌÀÆ & lr-AdamW/pg1‚0.0002722194900993462 & lr-AdamW/pg2‚0.0002722194900993462 trainer/global_step‚9249 % _timestamp‚1.7842806901227295e+09 _runtime‚7882.248422017 _step‚759÷꿦2Á- _wandbruntime‚7882  _runtime‚7882š™Só'ÂíO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9787463918592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85687886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159063064 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73375180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309705973557248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36137177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9772194398914 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9787122579456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1406395398037504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73375178 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319602214883840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9772534948804 % _timestamp‚1.7842806901238694e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16205091 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1408930429915842 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1416182861956096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19932357 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9787463918592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9787122579456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319602118180352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1416182861956096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1406395398037504 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1408930787505092 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85697863 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73375178 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19932089 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚7882.249634636 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85697863 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1416182503577600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9896163235328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73375180 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1416182503577600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36137448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9896241326592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1416182503577600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9896163235328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73375180 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1416182503577600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36137448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9896241326592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9772534948804 % _timestamp‚1.7842806901238694e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16205091 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1408930429915842 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1416182861956096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19932357 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9787463918592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9787122579456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319602118180352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1416182861956096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1406395398037504 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1408930787505092 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85697863 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73375178 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19932089 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚7882.249634636 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85697863øv& ^2Ä- _wandbruntime‚7882  _runtime‚7882¨Zˆlþ&ÅøM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319636575571456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36140907 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9897191867904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842806907354593e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9788476552704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73382812 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159089741 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73382810 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309739402315776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1409078309230798 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85696938 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36141173 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9788476552704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309739383703552 _runtime‚7882.861509355 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9897274146304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159089741 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1406542639067136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73382812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9773205452636 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16206833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319636676462080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚9249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1416331115619840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85706929 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1399304746188800 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1406542622027776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0£ý'ÆýM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9788476552704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73382812 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159089741 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73382810 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309739402315776 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1409078309230798 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85696938 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36141173 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9788476552704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309739383703552 _runtime‚7882.861509355 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9897274146304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159089741 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1406542639067136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73382812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9773205452636 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16206833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319636676462080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚9249 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1416331115619840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85706929 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1399304746188800 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1406542622027776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319636575571456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36140907 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9897191867904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842806907354593e+09ùQ”2Ç- _wandbruntime‚7882  _runtime‚7882.w¡´ È„ 3loss/FBt2/assignment_loss‚0.0001627489462934017 .loss/FRt2/detection_loss‚1.145190878652777 2loss/SRqqt2/assignment_loss‚0.4819075334593316 2loss/FBt1/assignment_loss‚0.024902303357234878 1loss/SRqqt1/detection_loss‚0.7929328305954186 /loss/FBt2/detection_loss‚0.9729107593496641 epoch‚23 _runtime‚7882.864005528 _step‚762 0loss/FRt1/assignment_loss‚1.5378878733296415 .loss/FRt1/detection_loss‚0.505048224225918 1loss/SRqqt2/detection_loss‚0.6212974922262523 /loss/FBt1/detection_loss‚0.3587321374621415 %loss/total_loss‚7.660650234699743 trainer/global_step‚9249 % _timestamp‚1.7842806907386692e+09 0loss/FRt2/assignment_loss‚0.8256744334516912 3loss/SRqqt1/assignment_loss‚0.39400301964337814ƒW¾É‰ 3loss/FBt2/assignment_loss‚0.0001627489462934017 .loss/FRt2/detection_loss‚1.145190878652777 2loss/SRqqt2/assignment_loss‚0.4819075334593316 2loss/FBt1/assignment_loss‚0.024902303357234878 1loss/SRqqt1/detection_loss‚0.7929328305954186 /loss/FBt2/detection_loss‚0.9729107593496641 epoch‚23 _runtime‚7882.864005528 _step‚762 0loss/FRt1/assignment_loss‚1.5378878733296415 .loss/FRt1/detection_loss‚0.505048224225918 1loss/SRqqt2/detection_loss‚0.6212974922262523 /loss/FBt1/detection_loss‚0.3587321374621415 %loss/total_loss‚7.660650234699743 trainer/global_step‚9249 % _timestamp‚1.7842806907386692e+09 0loss/FRt2/assignment_loss‚0.8256744334516912 3loss/SRqqt1/assignment_loss‚0.39400301964337814úÇÖR“Ê‚(:ˆ ùìçÒ†‰á¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$àøË‚(:í ùìçÒ¶ŽÐ¦ network.sent‚ 1.4704841e+07 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0! disk.rbd1.out‚103.50537109375# disk./.usageGB‚548.498348236084 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 293.96484375 disk.sda3.in‚0 disk.dm-1.out‚ 293.96484375$ memory_percent‚97.22270965576172 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚35779.7177734375 disk.sda3.out‚ 293.96484375) proc.memory.percent‚17.15383529663086 cpu‚26.06118645759233) disk./.usagePercent‚36.81018437089159 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 381.1640625 disk.sda.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚35779.7177734375 network.recv‚ 2.088725e+06% proc.memory.availableMB‚ 455.03125! proc.memory.rssMB‚ 2810.484375 disk.nvme0n1.in‚ 381.1640625% disk.nvme0n1.out‚35779.7177734375 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.dm-0.in‚ 381.1640625#†-Ý“Ì‚(:ˆ ˆíçÒùª‰¤ gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚151.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00o…ûÍ‚(:ð ˆíçÒ˜µ¦ disk.sda2.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 381.87890625 disk.sda.in‚0 disk.sda.out‚ 297.484375! disk.rbd1.out‚103.56787109375( proc.memory.availableMB‚ 455.21484375* disk./.usagePercent‚36.812187117831954# disk.dm-0.out‚35886.84912109375 disk.drbd2165.in‚0$ memory_percent‚97.22158908843994 disk.rbd3.out‚0 disk.dm-0.in‚ 381.87890625 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 297.484375 disk.dm-1.in‚0) proc.memory.percent‚17.15383529663086 cpu‚26.0605407025882 disk.rbd3.in‚0& disk.nvme0n1.out‚35886.84912109375( disk.nvme0c0n1.out‚35886.84912109375 proc.cpu.threads‚17 network.sent‚ 1.4757337e+07 network.recv‚ 2.093953e+06 disk.sda1.out‚0 disk.dm-1.out‚ 297.484375# disk./.usageGB‚548.528190612793 disk.nvme0n1.in‚ 381.87890625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625! proc.memory.rssMB‚ 2810.484375;¾Õ\“΂(:ˆ —íçÒîÕ¥¤ gpu.0.gpu‚81.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÂt•úÏ‚(:ê —íçÒŠü¶¦) disk./.usagePercent‚36.81198973602305 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0! disk.rbd1.out‚103.56787109375 network.recv‚ 2.097565e+06 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚ 299.5078125' proc.memory.availableMB‚ 454.9609375 cpu‚26.059890326934422 disk.sda1.in‚0 disk.sda.out‚ 299.5078125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 381.87890625# disk.nvme0n1.out‚35917.76953125 disk.sda2.out‚0 disk.dm-1.out‚ 299.5078125 disk.rbd3.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 381.87890625% disk.nvme0c0n1.out‚35917.76953125 disk.dm-0.in‚ 381.87890625! proc.memory.rssMB‚ 2810.484375) proc.memory.percent‚17.15383529663086*~÷û disk.sda2.in‚0 disk.rbd1.in‚0.0625# memory_percent‚97.2231388092041 disk.dm-0.out‚35917.76953125$ disk./.usageGB‚548.5252494812012 disk.rbd0.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.4760259e+07ò¢r»2Ð- _wandbruntime‚7922  _runtime‚7922ub ÇÑÁ & lr-AdamW/pg2‚0.0002703462748776421 trainer/global_step‚9299 % _timestamp‚1.7842807299994805e+09 _runtime‚7922.125068164 _step‚763 & lr-AdamW/pg1‚0.0002703462748776421oÇ|ÝÌÒÆ & lr-AdamW/pg1‚0.0002703462748776421 & lr-AdamW/pg2‚0.0002703462748776421 trainer/global_step‚9299 % _timestamp‚1.7842807299994805e+09 _runtime‚7922.125068164 _step‚763û†Õëñ2Ó- _wandbruntime‚7922  _runtime‚7922CtÓŸó'ÔíO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚321325181809664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16292341 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1416306937585600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1423595250561024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9823151435712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9837818081280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321325083009024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311377392871424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1423595250561024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _runtime‚7922.126566242 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86152315 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73756778 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73756778 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1413757449519104 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1406483769110528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1423595608939520 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1413757449519104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9947770326016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36328272 % _timestamp‚1.7842807300005012e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1423595608939520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73756780 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9837818081280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73756780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16292338 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9947690137600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86142338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20036201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 trainer/global_step‚9299K±Vø'ÕòO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚321325181809664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16292341 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1416306937585600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1423595250561024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9823151435712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9837818081280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321325083009024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311377392871424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1423595250561024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _runtime‚7922.126566242 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86152315 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73756778 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73756778 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1413757449519104 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1406483769110528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1423595608939520 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1413757449519104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9947770326016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36328272 % _timestamp‚1.7842807300005012e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1423595608939520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73756780 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9837818081280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73756780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16292338 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9947690137600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86142338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20036201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 trainer/global_step‚9299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508üL@.2Ö- _wandbruntime‚7922  _runtime‚7922jO"þ&×øM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159915800 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1413904673509376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9838830708224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 _step‚765 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86151390 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311410840242176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86151390 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9838830708224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚7922.737754373 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9948796551680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159915800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9823821939544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16294083 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321359636793856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1416454459311306 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73764412 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86161381 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36332071 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20037988 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1416454101721944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159925793 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1406630296821760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73764410 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159925793 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842807306119318e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1423743504217600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73764410 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73764412 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16294083 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321359636793856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1416454459311306 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73764412 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86161381 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36332071 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20037988 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactiv_\Ù0V!e_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1416454101721944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159925793 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1406630296821760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73764410 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159925793 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842807306119318e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1423743504217600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73764410 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73764412 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚7922.737754373 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9948796551680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159915800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9823821939544ýÿàÍ•2Ù- _wandbruntime‚7922  _runtime‚7922DM Ú† 1loss/SRqqt2/detection_loss‚0.5840632316341058 /loss/FBt2/detection_loss‚0.7689655671787002 trainer/global_step‚9299 % _timestamp‚1.7842807306150568e+09 0loss/FRt1/assignment_loss‚1.6585825553212141 epoch‚23 _runtime‚7922.740391331 2loss/SRqqt1/assignment_loss‚0.4204757306945125 /loss/FRt2/detection_loss‚1.2671833836220419 0loss/FBt1/detection_loss‚0.41565146785357904 %loss/total_loss‚7.882107358666274 _step‚766 0loss/FRt2/assignment_loss‚0.7792480061898726 3loss/FBt2/assignment_loss‚5.042733751188823e-05 /loss/FRt1/detection_loss‚0.4869128130959358 2loss/SRqqt2/assignment_loss‚0.4786047063955945 2loss/FBt1/assignment_loss‚0.015139075457034825 1loss/SRqqt1/detection_loss‚1.0072303938861709¼½I¸Û‹ 2loss/SRqqt1/assignment_loss‚0.4204757306945125 /loss/FRt2/detection_loss‚1.2671833836220419 0loss/FBt1/detection_loss‚0.41565146785357904 %loss/total_loss‚7.882107358666274 _step‚766 0loss/FRt2/assignment_loss‚0.7792480061898726 3loss/FBt2/assignment_loss‚5.042733751188823e-05 /loss/FRt1/detection_loss‚0.4869128130959358 2loss/SRqqt2/assignment_loss‚0.4786047063955945 2loss/FBt1/assignment_loss‚0.015139075457034825 1loss/SRqqt1/detection_loss‚1.0072303938861709 1loss/SRqqt2/detection_loss‚0.5840632316341058 /loss/FBt2/detection_loss‚0.7689655671787002 trainer/global_step‚9299 % _timestamp‚1.7842807306150568e+09 0loss/FRt1/assignment_loss‚1.6585825553212141 epoch‚23 _runtime‚7922.740391331þ¡íÅŸšÜ‚(: ¦íçÒ¶ñÈ¥ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂAKTÿÝ‚(:ô ¦íçÒŸ´º¨ disk.dm-1.in‚0" disk.dm-0.out‚35925.2607421875 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.rbd1.out‚103.69287109375 network.recv‚ 2.100755e+06% disk.nvme0n1.out‚35925.2607421875 proc.cpu.threads‚17 disk.dm-0.in‚ 382.70703125 disk.sda.out‚ 300.47265625 disk.sda2.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚35925.2607421875 disk.sda2.out‚0 cpu‚26.059262755569357 network.sent‚ 1.477834e+07 disk.rbd3.in‚0 disk.nvme0n1.in‚ 382.70703125 disk.rbd0.out‚0 disk.sda3.out‚ 300.47265625' proc.memory.availableMB‚ 454.9765625) proc.memory.percent‚17.15383529663086* disk./.usagePercent‚36.811973607548396" disk.nvme0c0n1.in‚ 382.70703125$ memory_percent‚97.22304344177246$ disk./.usageGB‚548.5250091552734 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 300.47265625 disk.rbd1.in‚0.0625! proc.memory.rssMB‚ 2810.484375 disk.sda1.in‚0 disk.sda.in‚0¥¬ “Þ‚(:ˆ µíçÒêù¤ gpu.0.gpu‚84.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø­²ìß‚(:á µíçÒ¡á­§& disk.nvme0n1.out‚35936.41943359375 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚300.875 disk.rbd3.out‚0$ memory_percent‚97.22414016723633 disk.sda3.out‚300.875 disk.rbd3.in‚0! proc.memory.rssMB‚ 2810.484375* disk./.usagePercent‚36.811977959676476# disk./.usageGB‚548.525074005127 disk.sda1.in‚0 network.sent‚ 1.4828282e+07 disk.sda2.out‚0 disk.dm-0.in‚ 383.421875& proc.memory.availableMB‚ 454.796875( disk.nvme0c0n1.out‚35936.41943359375 disk.rbd0.in‚0 disk.rbd1.in‚0.0625 cpu‚26.058627487832016 proc.cpu.threads‚17 disk.sda.out‚300.875# disk.dm-0.out‚35936.41943359375 disk.nvme0c0n1.in‚ 383.421875 network.recv‚ 2.105745e+06 disk.sda.in‚0 disk.nvme0n1.in‚ 383.421875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0! disk.rbd1.out‚103.69287109375) proc.memory.percent‚17.15383529663086‹V¿2à- _wandbruntime‚7962  _runtime‚7962(ÏzðÇáÁ & lr-AdamW/pg2‚0.0002684708751179498 trainer/global_step‚9349 % _timestamp‚1.7842807699100459e+09 _runtime‚7962.035675292 _step‚767 & lr-AdamW/pg1‚0.0002684708751179498@¯ÿ²ÌâÆ & lr-AdamW/pg1‚0.0002684708751179498 & lr-AdamW/pg2‚0.0002684708751179498 trainer/global_step‚9349 % _timestamp‚1.7842807699100459e+09 _runtime‚7962.035675292 _step‚767ÿÿvÏõ2ã- _wandbruntime‚7962  _runtime‚7962CÆ95ó'äíO trainer/global_step‚9349 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86596790 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9888513583104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1423682730076858 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚7962.040367721 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160735168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86596790 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9999120336384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9999200524800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9888513583104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86606767 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1413809319743488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160745147 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74138378 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1421119501000704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1421119483961344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20139909 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9873427372730 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1431008355922944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313048849410048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9888854922240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1423683087666108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9873767922620 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86606767 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323047951134208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313048830797824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74138378 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9888854922240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842807699111152e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1431007997544448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323048049934848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1431007997544448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74138380 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36519500 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20139656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3575892503‹×£÷"åòO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1421119501000704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1421119483961344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20139909 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9873427372730 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1431008355922944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313048849410048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9888854922240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1423683087666108 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9873767922620 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86606767 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323047951134208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313048830797824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74138378 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9888854922240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842807699111152e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1431007997544448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323048049934848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1431007997544448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74138380 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36519500 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20139656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚9349 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86596790 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9888513583104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1423682730076858 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚7962.040367721 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160735168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86596790 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9999120336384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9999200524800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 55475814?xæ4 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9888513583104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86606767 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1413809319743488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160745147 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74138378 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088€cïd*2æ- _wandbruntime‚7962  _runtime‚79623ÀîR'çúM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36522989 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1423830609391814 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74146012 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74146010 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20141915 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1413955813376000 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10000147257344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160761845 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9889526210048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9874778976454 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86605842 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9889526210048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9874778976454 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86605842 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36522989 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1423830609391814 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74146012 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74146010 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20141915 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1413955813376000 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10000147257344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160761845 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_'¶FÃ¥train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1431058880157366 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74519978 trainer/global_step‚9399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87061219 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1431059237746616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9939550424064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36709927 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74519980 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9924384409528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1438420744527872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1421134853337088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87051242 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1421134836297728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842808107018247e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1428481552482304 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1428481552482304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924043859638 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36710201 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20243360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161581199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9939550424064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74519978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10050633193984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚8002.827633341 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1431058880157366 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74519978 trainer/global_step‚9399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87061219 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1431059237746616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9939550424064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36709927 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74519980 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9924384409528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1438420744527872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1421134853337088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87051242 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1421134836297728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842808107018247e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1428481552482304 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1428481552482304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924043859638 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36710201 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20243360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161581199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9939550424064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74519978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10050633193984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚8002.827633341 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324805285289472 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1438569356570112 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1428628776472576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16468583 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1438569356570112 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161597897 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1428628776472576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9940221711872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1421281346969600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74527610 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9940563058176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20245375 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1421281364008960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9940221711872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74527612 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36713697 % _timestamp‚1.7842808113420908e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87070285 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1428628793511936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9940563058176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1431206401882960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161597897 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1428628793511936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36713961v/t úþM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161587904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16468586 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314753697482752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161587904 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20245114 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1438568998184448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87070285 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚773 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1431206759472322 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1428628793511936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9940563058176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1431206401882960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161597897 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1428628793511936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36713961 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10051587806720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324805285289472 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1438569356570112 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1428628776472576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16468583 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1438569356570112 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161597897 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1428628776472576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9940221711872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J1'.¶÷ DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1421281346969600 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74527610 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9940563058176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20245375 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1421281364008960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9940221711872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74527612 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36713697 % _timestamp‚1.7842808113420908e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87070285 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1445833491511296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10101949343232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1435843586924544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9989904586752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9989904586752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74901580 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16554091 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1428460369891328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1438435030237874 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9990245925888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74901580 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36900674 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87505694 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1428460386930688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9974660346546 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚8045.782663163 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1435843586924544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10102037920256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1445833849889792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162417251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87505694 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316391706650624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74901578 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1445833491511296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16554088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326493655993856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9975000896436 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20346865 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0EŽÛâû'‰õO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16554091 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1428460369891328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1438435030237874 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9990245925888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74901580 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36900674 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87505694 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1428460386930688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9974660346546 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚8045.782663163 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1435843586924544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10102037920256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1445833849889792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162417251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87505694 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316391706650624 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74901578 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1445833491511296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16554088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326493655993856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9975000896436 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20346865 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1445833491511296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10101949343232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1435843586924544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9989904586752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9989904586752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74901580 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ˆŒã2Š- _wandbruntime‚8046  _runtime‚8046€jóÝs ‹ùM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16555833 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1428606880563200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87524737 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1435990827954176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20348647 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1438582551963468 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316425154021376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162433949 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceSta÷ ÂŒtsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74909212 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36904734 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9975671400268 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1445982103553536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1435990844993536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9990917213696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16555836 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9990917213696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9991258560000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326528216258560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74909210 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10103062237184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9976011950270 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74909210 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1438582909552830 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1435990844993536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87514746 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10102986250240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162433949 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162423956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20348898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9991258560000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36904480 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚8046.415020051 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991F<Ÿ¨'ŒþM [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1428606880563200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87524737 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1435990827954176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20348647 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1438582551963468 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316425154021376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162433949 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74909212 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36904734 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9975671400268 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1445982103553536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1435990844993536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9990917213696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16555836 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9990917213696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9991258560000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326528216258560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74909210 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10103062237184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9976011950270 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74909210 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1438582909552830 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1435990844993536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87514746 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10102986250240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162433949 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162423956 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20348898 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9991258560000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36904480 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚8046.415020051 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16555833 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983‰w"b 2- _wandbruntime‚8046  _runtime‚8046:j° Žƒ 0loss/FRt1/assignment_loss‚1.6796355762527984 1loss/SRqqt2/assignment_loss‚0.500661125058379 0loss/FBt1/detection_loss‚0.41333030317550495 epoch‚23 trainer/global_step‚9449 % _timestamp‚1.7842808542911024e+09 /loss/FRt2/assignment_loss‚0.821867215181516 2loss/FBt1/assignment_loss‚0.005262419009413253 /loss/FRt2/detection_loss‚1.2055771080119848 1loss/SRqqt2/detection_loss‚0.6278429643954548 /loss/FBt2/detection_loss‚0.8583028524127954 _step‚778 2loss/SRqqt1/assignment_loss‚0.4757240367883864 1loss/SRqqt1/detection_loss‚0.8758459639992859 %loss/total_loss‚7.990741627394083 _runtime‚8046.416518751 2loss/FBt2/assignment_loss‚0.020511618100793657 /loss/FRt1/detection_loss‚0.5061804450077705wŸjˆ %loss/total_loss‚7.990741627394083 _runtime‚8046.416518751 2loss/FBt2/assignment_loss‚0.020511618100793657 /loss/FRt1/detection_loss‚0.5061804450077705 0loss/FRt1/assignment_loss‚1.6796355762527984 1loss/SRqqt2/assignment_loss‚0.500661125058379 0loss/FBt1/detection_loss‚0.41333030317550495 epoch‚23 trainer/global_step‚9449 % _timestamp‚1.7842808542911024e+09 /loss/FRt2/assignment_loss‚0.821867215181516 2loss/FBt1/assignment_loss‚0.005262419009413253 /loss/FRt2/detection_loss‚1.2055771080119848 1loss/SRqqt2/detection_loss‚0.6278429643954548 /loss/FBt2/detection_loss‚0.8583028524127954 _step‚778 2loss/SRqqt1/assignment_loss‚0.4757240367883864 1loss/SRqqt1/detection_loss‚0.8758459639992859Š‚ù……‘‚(:† žîçÒÍÝ°¥ gpu.0.gpu‚75.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶áLùê‘‚(:ß žîçÒÔ†© network.sent‚ 1.4978537e+07) disk./.usagePercent‚36.82809593802471 disk.sda1.in‚0 disk.rbd3.out‚0 network.recv‚ 2.132755e+06 disk.sda1.out‚0 disk.rbd0.out‚0! disk.nvme0c0n1.out‚ 36301.9375! disk.rbd1.out‚103.94287109375 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 387.3359375 disk.sda2.in‚0 disk.sda3.out‚ 308.84765625$ disk./.usageGB‚548.7652435302734 disk.rbd0.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.nvme0n1.out‚ 36301.9375 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625! proc.memory.rssMB‚ 2810.484375 disk.dm-0.in‚ 387.3359375 disk.dm-0.out‚ 36301.9375 disk.nvme0n1.in‚ 387.3359375 disk.drbd2165.out‚0 disk.dm-1.out‚ 308.84765625( proc.memory.availableMB‚ 454.73828125 cpu‚26.05405885159234 disk.sda.out‚ 308.84765625 disk.sda2.out‚0 disk.rbd3.in‚0$ memory_percent‚97.22449779510498) proc.memory.percent‚17.15383529663086õ<¬“’‚(:ˆ ­îçÒš·ˆ¤ gpu.0.gpu‚85.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚251.037' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.2074 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 æî“‚(:ã ­îçÒ׋¦ disk.rbd0.in‚0 disk.sda.out‚ 309.41796875 disk.sda3.in‚0 disk.dm-0.out‚ 36311.21875 network.sent‚ 1.5043598e+07 network.recv‚ 2.137878e+06 disk.nvme0n1.out‚ 36311.21875 disk.sda.in‚0 disk.sda3.out‚ 309.41796875 proc.cpu.threads‚17) disk./.usagePercent‚36.82844231621854 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625) proc.memory.percent‚17.15383529663086$ disk./.usageGB‚548.7704048156738 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 388.05078125 disk.rbd3.out‚0 disk.dm-0.in‚ 388.05078125 cpu‚26.05345637740957 disk.sda1.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 388.05078125 disk.dm-1.out‚ 309.41796875! proc.memory.rssMB‚ 2810.484375" disk.nvme0c0n1.out‚ 36311.21875! disk.rbd1.out‚103.94287109375$ memory_percent‚97.22261428833008& proc.memory.availableMB‚ 455.046875êá@“”‚(:ˆ ¼îçÒæÉÕ¤ gpu.0.gpu‚36.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tuqó•‚(:è ¼îçÒ‚½Þ¦ network.sent‚ 1.5046621e+07 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚309.6875! disk.rbd1.out‚104.06787109375! proc.memory.rssMB‚ 2810.484375) disk./.usagePercent‚36.82844590032402 disk.sda.out‚309.6875 disk.dm-0.in‚ 388.87890625 network.recv‚ 2.141223e+06" disk.dm-0.out‚36318.1279296875$ memory_percent‚97.22521305084229 cpu‚26.05279993278241 disk.nvme0n1.in‚ 388.87890625' disk.nvme0c0n1.out‚36318.1279296875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 454.62109375) proc.memory.percent‚17.15383529663086$ disk./.usageGB‚548.7704582214355 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚36318.1279296875 disk.rbd0.in‚0 disk.sda.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 388.87890625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚309.6875ÁÀ]2–- _wandbruntime‚8088  _runtime‚8088Œ'¼Ç—Á _runtime‚8088.294375927 _step‚779 & lr-AdamW/pg1‚0.0002628327738002812 & lr-AdamW/pg2‚0.0002628327738002812 trainer/global_step‚9499 % _timestamp‚1.7842808961682541e+09…ЈvÌ˜Æ trainer/global_step‚9499 % _timestamp‚1.7842808961682541e+09 _runtime‚8088.294375927 _step‚779 & lr-AdamW/pg1‚0.0002628327738002812 & lr-AdamW/pg2‚0.0002628327738002812‹$lФ2™- _wandbruntime‚8088  _runtime‚8088ÈkЕ>$šöO ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1443205638406144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1453246238494720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚8088.295194833 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87960146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87960146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75283178 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1435785920524288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚9499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1443205638406144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37091847 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75283178 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87970123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16641338 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328216632487936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10153374430208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10040600088576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87970123 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163253303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1453246596873216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75283180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842808961692386e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S½Ìj¾>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1453246238494720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10040941427712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318063144577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10040941427712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16641341 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20450774 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101hˆÿö(›ûO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16641338 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328216632487936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10153374430208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10040600088576 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87970123 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163253303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1453246596873216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75283180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842808961692386e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1453246238494720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10040941427712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318063144577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10040941427712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16641341 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20450774 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1443205638406144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1453246238494720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚8088.295194833 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87960146 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87960146 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75283178 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1435785920524288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚9499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1443205638406144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37091847 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75283178 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87970123Œ¸õ{{2œ- _wandbruntime‚8088  _runtime‚8088b÷\'€N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163270001 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75290812 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318096591947776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16643083 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10041954061824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1445959059633338 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1453394492151296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163260008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10026287887176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20452873 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163270001 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1443352879435776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163260008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16643086 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1453394850536960 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1443352879435776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10154490649600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1435932431196160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75290810 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1445958702043976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1443352896475136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87969198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87979189 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842808968043551e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87979189 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20452606 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10041954061824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10041612715520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328250977512448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10154404176896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚9499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37095689 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1453394850536960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _runtime‚8088.930355667 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1443352896475136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚9499 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37095689 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1453394850536960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _runtime‚8088.930355667 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1443352896475136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16643083 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10041954061824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1445959059633338 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1453394492151296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163260008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10026287887176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20452873 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163270001 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1443352879435776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163260008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16643086 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1453394850536960 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1443352879435776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10154490649600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1435932431196160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75290810 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1445958702043976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1443352896475136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87969198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87979189 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842808968043551e+09Ì‚â12Ÿ- _wandbruntime‚8088  _runtime‚8088cT¹  … /loss/FRt1/detection_loss‚0.4990989813175233 1loss/SRqqt1/detection_loss‚0.8324247580473527 /loss/FBt2/detection_loss‚0.7696602566597553 trainer/global_step‚9499 _step‚782 0loss/FRt1/assignment_loss‚1.6208992187645057 2loss/SRqqt1/assignment_loss‚0.4227672702064537 _runtime‚8088.931810875 0loss/FRt2/assignment_loss‚0.8665320548658353 3loss/SRqqt2/assignment_loss‚0.47874670890763776 1loss/FBt1/assignment_loss‚0.03471773103069188 3loss/FBt2/assignment_loss‚0.0005714325797262413 /loss/FRt2/detection_loss‚1.2902448521682903 1loss/SRqqt2/detection_loss‚0.6588564288040769 0loss/FBt1/detection_loss‚0.40101650913895454 %loss/total_loss‚7.875536202490803 $ _timestamp‚1.784280896806391e+09 epoch‚23oœ—z‰¡Š epoch‚23 /loss/FRt1/detection_loss‚0.4990989813175233 1loss/SRqqt1/detection_loss‚0.8324247580473527 /loss/FBt2/detection_loss‚0.7696602566597553 trainer/global_step‚9499 _step‚782 0loss/FRt1/assignment_loss‚1.6208992187645057 2loss/SRqqt1/assignment_loss‚0.4227672702064537 _runtime‚8088.931810875 0loss/FRt2/assignment_loss‚0.8665320548658353 3loss/SRqqt2/assignment_loss‚0.47874670890763776 1loss/FBt1/assignment_loss‚0.03471773103069188 3loss/FBt2/assignment_loss‚0.0005714325797262413 /loss/FRt2/detection_loss‚1.2902448521682903 1loss/SRqqt2/detection_loss‚0.6588564288040‰p–°‡769 0loss/FBt1/detection_loss‚0.40101650913895454 %loss/total_loss‚7.875536202490803 $ _timestamp‚1.784280896806391e+09Žêf’ñ“¢‚(:ˆ ËîçÒŽ’¸¥ gpu.0.gpu‚83.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?oèö£‚(:ë ËîçÒ¦­ý§ disk.sda.in‚0& proc.memory.availableMB‚ 454.640625! proc.memory.rssMB‚ 2810.484375 cpu‚26.052234178025998 disk.sda3.out‚ 310.07421875! disk.rbd1.out‚104.06787109375 disk.rbd3.out‚0# disk.dm-0.out‚36327.47705078125 disk.sda.out‚ 310.07421875 disk.rbd0.in‚0 network.recv‚ 2.14507e+06 disk.sda3.in‚0) disk./.usagePercent‚36.82843284393977 disk.drbd2165.in‚0$ memory_percent‚97.22509384155273) proc.memory.percent‚17.15383529663086 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 389.59375( disk.nvme0c0n1.out‚36327.47705078125 network.sent‚ 1.5064694e+07 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 389.59375 disk.sda2.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 310.07421875 disk.dm-0.in‚ 389.59375 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚36327.47705078125# disk./.usageGB‚548.770263671875ä½ðœ¤‚(:„ ÚîçÒÖ£¢¥ gpu.0.gpu‚6.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆþÇa(:ã ÚîçÒ鉞§$ disk./.usageGB‚548.7692184448242 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 390.421875 disk.nvme0c0n1.in‚ 390.421875 disk.sda2.out‚0" disk.dm-0.out‚36335.4912109375 disk.rbd0.in‚0) disk./.usagePercent‚36.82836269787539 disk.rbd3.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.15383529663086 proc.cpu.threads‚17 network.recv‚ 2.150134e+06' disk.nvme0c0n1.out‚36335.4912109375 disk.sda.in‚0 disk.sda3.out‚310.3125 disk.dm-1.in‚0! proc.memory.rssMB‚ 2810.484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚310.3125$ memory_percent‚97.22654819488525 network.sent‚ 1.5116838e+07( proc.memory.availableMB‚ 454.40234375! disk.rbd1.out‚104.13037109375 disk.rbd3.out‚0 disk.dm-0.in‚ 390.421875% disk.nvme0n1.out‚36335.4912109375 disk.drbd2165.in‚0 disk.rbd0.out‚0 cpu‚26.051591524494008 disk.sda3.in‚0 disk.dm-1.out‚310.3125°ÔǶ“¦‚(:ˆ éîçÒ©öð¤ gpu.0.gpu‚36.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í³ÜŒ÷§‚(:ì éîçÒ–º¤§ disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚0) proc.memory.percent‚17.15383529663086 disk.nvme0n1.in‚ 390.421875 disk.rbd3.out‚0$ disk./.usageGB‚548.7721290588379% disk.nvme0n1.out‚36344.6044921875 disk.sda.in‚0 disk.sda3.out‚ 312.01171875 disk.dm-0.in‚ 390.421875' proc.memory.availableMB‚ 454.2109375 proc.cpu.threads‚17 disk.sda.out‚ 312.01171875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0! disk.rbd1.out‚104.13037109375 network.sent‚ 1.511976e+07 network.recv‚ 2.153585e+06) disk./.usagePercent‚36.82855803162401 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 390.421875 disk.dm-1.out‚ 312.01171875! proc.memory.rssMB‚ 2810.484375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0$ memory_percent‚97.22771644592285 cpu‚26.05097673297218' disk.nvme0c0n1.out‚36344.6044921875" disk.dm-0.out‚36344.6044921875(«s2¨- _wandbruntime‚8131  _runtime‚8131ë—Æ©À $ _timestamp‚1.784280939279388e+09 _runtime‚8131.404946565 _step‚783 & lr-AdamW/pg1‚0.0002609498417098469 & lr-AdamW/pg2‚0.0002609498417098469 trainer/global_step‚9549”lI˪Š& lr-AdamW/pg1‚0.0002609498417098469 & lr-AdamW/pg2‚0.0002609498417098469 trainer/global_step‚9549 $ _timestamp‚1.784280939279388e+09 _runtime‚8131.404946565 _step‚783|_292«- _wandbruntime‚8131  _runtime‚8131nT|ü'¬öO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164079376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75664780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319734601115648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88424575 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1450567689887744 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1450567689887744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10075893320362 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88414598 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10091295590400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10091636929536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1460658985478144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88424575 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚319734582503424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20554077 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1453187330398890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10076233870252 trainer/global_step‚9549 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37282933 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842809392804646e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37282665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16728588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164089355 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1443111454117888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚8131.409430902 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1460659343856640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1460659343856640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269yèTO(­ûO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88424575 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚319734582503424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20554077 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1453187330398890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10076233870252 trainer/global_step‚9549 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37282933 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842809392804646e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37282665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16728588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164089355 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1443111454117888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚8131.409430902 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1460659343856640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1460659343856640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164079376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75664780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319734601115648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88424575 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1450567689887744 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1450567689887744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10075893320362 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88414598 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10091295590400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10091636929536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1460658985478144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508Ú9Ð2®- _wandbruntime‚8132  _runtime‚8132á3öE\¯€N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88423650 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1460807239134720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1453335209713846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10092308217344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16730336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164106053 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10092649563648 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10092308217344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37286455 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75672410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20556382 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329973806366208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329973907256832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10205795104256 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1443257947750400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20556122 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1450714930917376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88423650 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88433641 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10076904374084 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1450714930917376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164106053 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1460807597520384 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842809399237454e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75672410 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1453334852124484 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16730333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319768029874176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10077244924086 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚8132.049810404 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164096060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 Æ '°…N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚9549 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842809399237454e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75672410 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1453334852124484 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16730333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319768029874176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10077244924086 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚8132.049810404 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164096060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88423650 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1460807239134720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1453335209713846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10092308217344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16730336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164106053 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10092649563648 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10092308217344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37286455 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75672410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20556382 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329973806366208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329973907256832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10205795104256 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1443257947750400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20556122 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1450714930917376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88423650 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88433641 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10076904374084 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1450714930917376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164106053 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1460807597520384 Q¹ü'¼öO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76046378 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7842809821007576e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10126509807270 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468072090840064 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1468072090840064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚788 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10126850357160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468071732461568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1457929741369344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚321406020429824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1457929758408704 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1457929758408704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16815841 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331662025741312 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1450436970672128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1460563480479398 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76046380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37473629 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88869050 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468071732461568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10142332431360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10141991092224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164925407 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7842809821007576e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10126509807270 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468072090840064 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1468072090840064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚788 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10126850357160 UGDeviceStatsMonitor.on_train_batch_start/reques“i´……ted_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468071732461568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1457929741369344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚321406020429824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1457929758408704 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1457929758408704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16815841 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331662025741312 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1450436970672128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1460563480479398 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76046380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37473629 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88869050 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468071732461568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10142332431360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10141991092224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164925407 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20660076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16817586 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20659832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164942105 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468220344503808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10127520860992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88878102 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321439449188352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10143003719168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164932112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76054010 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1460711359794354 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88888093 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1458076982398976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10143345065472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1450583481344000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 $ _timestamp‚1.784280982726763e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10257125297664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10143345065472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164942105 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37477415 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1460711002204992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10127861410994 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37477415 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1460711002204992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10127861410994 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20660076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16817586 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20659832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164942105 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468220344503808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10127520860992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88878102 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321439449188352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10143003719168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164932112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76054010 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1460711359794354 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88888093 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1458076982398976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10143345065472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1450583481344000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 $ _timestamp‚1.784280982726763e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10257125297664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10143345065472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164942105 E ɲ{ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951964020729065 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38962599635124207 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5213258266448975 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5026806592941284 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6525102853775024 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03773186728358269 7validation_average_jet_accuracy‚0.39486217498779297 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22775901854038239 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7523986101150513 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7589081525802612 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5871221423149109 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13683727383613586 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚791 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6392887830734253 jet/accuracy_5_of_5‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969227910041809 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.547827959060669 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29775509238243103 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 +particle/specificity‚0.9066815376281738 trainer/global_step‚9599 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9896136522293091 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6275032758712769 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_4_of_5‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 'loss/val_total_loss‚7.9368815013703 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7490390539169312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7379361391067505 7 Purity/0×5÷_z3FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8606986999511719 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24490129947662354 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.055190250277519226 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2240673452615738 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4895833432674408 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3065855801105499 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959391474723816 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7896069288253784 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877076148986816 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_3_of_3‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3083900213241577 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8617157936096191 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1514483392238617 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030421728268265724 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860448241233826 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8212396502494812 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6052872538566589 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7350322604179382 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 +jet/accuracy_2_of_2‚0.16032128036022186 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031081506982445717 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28527823090553284 *jet/accuracy_1_of_3‚0.9206733703613281 % _timestamp‚1.7842810009808054e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980698823928833 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0278488676995039 *jet/accuracy_1_of_4‚0.9785693883895874 'particle/f_score‚0.6276569366455078 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7727096080780029 /particle/accuracy_4_of_4‚0.9414735436439514 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.709017813205719 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962906837463379 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8117159008979797 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23009811341762543 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7755480408668518 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5709648132324219 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2533053755760193 )jet/accuracy_1_of_2‚0.796972930431366 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6593764424324036 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9278069734573364 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30808600783348083 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2873454689979553 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6519098281860352 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7077909111976624 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25621217489242554 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7474144101142883 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48994383215904236 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6564732789993286 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5062376856803894 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23350541293621063 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8119992613792419 *jet/accuracy_2_of_4‚0.8117786645889282 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2954275906085968 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955142140388489 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7393234968185425 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07251140475273132 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8437111973762512 particle/accuracy_5_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29775509238243103 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13440172374248505 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030778612941503525 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8673882484436035 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04873178154230118 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.739775538444519 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4348154067993164 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030613508075475693 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2683791518211365 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_1_of_3‚1 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0278488676995039 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 particle/accuracy_3_of_5‚1 _runtime‚8193.107168438 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29307854175567627 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8617157936096191 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6695014238357544 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21436607837677002 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3600669801235199 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23212648928165436 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03839166834950447 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7955834269523621 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7053200602531433 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24152950942516327 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7200474739074707 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6947110295295715 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7160360813140869 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7416565418243408 particle/accuracy_2_of_2‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994408369064331 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9087052941322327 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6639996767044067 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3264700174331665 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23796288669109344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49551692605018616 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3507879674434662 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977676868438721 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6640058755874634 particle/accuracy_1_of_1‚1 epoch‚23 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6670647859573364 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6593764424324036 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1201961413025856 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21833333373069763 +jet/accuracy_1_of_1‚0.35864219069480896 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.029217060655355453 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *particle/sensitivity‚0.613450288772583 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5853604674339294 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7503843903541565 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5854166746139526 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7785698771476746 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07256050407886505 +jet/accuracy_3_of_4‚0.07865994423627853 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6922662854194641 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6861585378646851 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870534539222717 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.391183078289032 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6654447913169861 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.703125 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6448933482170105 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22069483995437622 ,jet/accuracy_4_of_4‚0.003968254197388887 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6957353949546814 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837$ß¾p½=Ê·{ =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6957353949546814 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951964020729065 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38962599635124207 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5213258266448975 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5026806592941284 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6525102853775024 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03773186728358269 7validation_average_jet_accuracy‚0.39486217498779297 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22775901854038239 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7523986101150513 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7589081525802612 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5871221423149109 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13683727383613586 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚791 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6392887830734253 jet/accuracy_5_of_5‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969227910041809 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.547827959060669 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29775509238243103 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 +particle/specificity‚0.9066815376281738 trainer/global_step‚9599 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9896136522293091 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6275032758712769 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_4_of_5‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 'loss/val_total_loss‚7.9368815013703 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7490390539169312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7379361391067505 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8606986999511719 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24490129947662354 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.055190250277519226 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994983971118927 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2240673452615738 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4895833432674408 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3065855801105499 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959391474723816 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7896069288253784 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877076148986816 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_3_of_3‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3083900213241577 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8617157936096191 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1514483392238617 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030421728268265724 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860448241233826 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8212396502494812 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6052872538566589 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7350322604179382 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 +jet/accuracy_2_of_2‚0.16032128036022186 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031081506982445717 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28527823090553284 *jet/accuracy_1_of_3‚0.9206733703613281 % _timestamp‚1.7842810009808054e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980698823928833 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0278488676995039 *jet/accuracy_1_of_4‚0.9785693883895874 'particle/f_score‚0.6276569366455078 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7727096080780029 /particle/accuracy_4_of_4‚0.9414735436439514 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.709017813205719 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962906837463379 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8117159008979797 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23009811341762543 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7755480408668518 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5709648132324219 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2533053755760193 )jet/accuracy_1_of_2‚0.796972930431366 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6593764424324036 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9278069734573364 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30808600783348083 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2873454689979553 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6519098281860352 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7077909111976624 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25621217489242554 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7474144101142883 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48994383215904236 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6564732789993286 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5062376856803894 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23350541293621063 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8119992613792419 *jet/accuracy_2_of_4‚0.8117786645889282 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2954275906085968 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955142140388489 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7393234968185425 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07251140475273132 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8437111973762512 particle/accuracy_5_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29775509238243103 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13440172374248505 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030778612941503525 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8673882484436035 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04873178154230118 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.739775538444519 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4348154067993164 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030613508075475693 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2683791518211365 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_1_of_3‚1 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0278488676995039 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 particle/accuracy_3_of_5‚1 _runtime‚8193.107168438 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29307854175567627 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8617157936096191 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6695014238357544 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21436607837677002 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3600669801235199 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23212648928165436 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03839166834950447 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7955834269523621 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7053200602531433 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24152950942516327 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7200474739074707 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6947110295295715 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7160360813140869 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7416565418243408 particle/accuracy_2_of_2‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994408369064331 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9087052941322327 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6639996767044067 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3264700174331665 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23796288669109344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49551692605018616 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3507879674434662 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977676868438721 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6640058755874634 particle/accuracy_1_of_1‚1 epoch‚23 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6670647859573364 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6593764424324036 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1201961413025856 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21833333373069763 +jet/accuracy_1_of_1‚0.35864219069480896 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.029217060655355453 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *particle/sensitivity‚0.613450288772583 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5853604674339294 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7503843903541565 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5854166746139526 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7785698771476746 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07256050407886505 +jet/accuracy_3_of_4‚0.07865994423627853 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6922662854194641 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6861585378646851 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9870534539222717 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.391183078289032 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6654447913169861 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.703125 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6448933482170105 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22069483995437622 ,jet/accuracy_4_of_4‚0.003968254197388887 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0—F›% “Ë‚(:ˆ ´ïçÒ–抣 gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹¨âûÌ‚(:ð ´ïçÒ»ó‰¥ disk.rbd1.in‚0.0625 disk.drbd2165.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚36.82818963678224( disk.nvme0c0n1.out‚36476.44873046875 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 393.5078125 disk.nvme0n1.in‚ 393.5078125 disk.drbd2165.out‚0* proc.memory.percent‚17.156696319580078 disk.sda3.out‚ 317.4765625 disk.sda1.in‚0 network.sent‚ 1.526121e+07 disk.rbd3.in‚0# disk.dm-0.out‚36476.44873046875 disk.rbd0.in‚0& proc.memory.availableMB‚ 452.796875! proc.memory.rssMB‚ 2810.953125$ disk./.usageGB‚548.7666397094727 disk.sda.out‚ 317.4765625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 317.4765625! disk.rbd1.out‚104.31787109375 disk.sda3.in‚0 disk.rbd3.out‚0 network.recv‚ 2.174353e+06 disk.sda1.out‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 393.5078125& disk.nvme0n1.out‚36476.44873046875$ memory_percent‚97.23634719848633 cpu‚26.056285036116922s‘f­“Í‚(:ˆ ÃïçÒµÎ×¢ gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ô”‹Î‚(:ö ÃïçÒ©ñ¤ disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚97.23482131958008) disk./.usagePercent‚36.82819091681991# disk./.usageGB‚548.766658782959! disk.rbd1.out‚104.31787109375 network.recv‚ 2.179279e+06 disk.nvme0n1.in‚ 394.22265625 disk.rbd0.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚36486.26904296875 disk.rbd0.in‚0 disk.sda.out‚ 317.79296875 disk.rbd3.out‚0 disk.dm-0.in‚ 394.22265625( disk.nvme0c0n1.out‚36486.26904296875 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 cpu‚26.055639732699248 network.sent‚ 1.5318828e+07 disk.drbd2165.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 317.79296875& proc.memory.availableMB‚ 453.046875* proc.memory.percent‚17.156696319580078 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 317.79296875# disk.dm-0.out‚36486.26904296875" disk.nvme0c0n1.in‚ 394.22265625 disk.rbd3.in‚0! proc.memory.rssMB‚ 2810.953125 proc.cpu.threads‚17r£´üžÏ‚(:“ ÒïçÒ¬ÚÊ¢ gpu.0.gpu‚72.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚185.049' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.009800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±m'òÑЂ(:ñ ÒïçÒÙÌÂ¥ cpu‚26.055051963492843 disk.sda.in‚0 disk.rbd3.in‚0( disk./.usagePercent‚36.8281863086843 disk.rbd0.out‚0' disk.nvme0c0n1.out‚36493.2763671875 disk.sda3.in‚0 disk.dm-1.out‚ 317.9765625' proc.memory.availableMB‚ 453.2421875 disk.sda2.out‚0 disk.dm-0.in‚ 395.05078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 395.05078125 disk.sda1.out‚0! proc.memory.rssMBðÜ»+‚ 2810.953125$ disk./.usageGB‚548.7665901184082" disk.nvme0c0n1.in‚ 395.05078125 disk.rbd1.in‚0.0625! disk.rbd1.out‚104.44287109375" disk.dm-0.out‚36493.2763671875% disk.nvme0n1.out‚36493.2763671875 network.recv‚ 2.182611e+06 disk.sda.out‚ 317.9765625 disk.sda3.out‚ 317.9765625 disk.sda1.in‚0$ memory_percent‚97.23362922668457 network.sent‚ 1.5323873e+07 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.156696319580078­OwR2Ñ- _wandbruntime‚8235  _runtime‚8235´ÐÉÒà _runtime‚8235.692633219 _step‚792 ' lr-AdamW/pg1‚0.00025717935549980045 ' lr-AdamW/pg2‚0.00025717935549980045 trainer/global_step‚9649 % _timestamp‚1.7842810435669734e+09 bî>ÎÓÈ ' lr-AdamW/pg2‚0.00025717935549980045 trainer/global_step‚9649 % _timestamp‚1.7842810435669734e+09 _runtime‚8235.692633219 _step‚792 ' lr-AdamW/pg1‚0.00025717935549980045˜1ÖÜ2Ô- _wandbruntime‚8235  _runtime‚82357½\ü'ÕöO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323093389068288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1477432137679360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10325435584512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76548644 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10195186569297 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89393824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1459671167479808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚8235.697425197 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89403801 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10210419799552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16905608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7842810435681603e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20793363 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165942468 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333418824652800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333418719560704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10194846019407 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1469866371088465 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165952447 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1477431779300864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37698700 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76548646 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269οÓ(ÖûO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1477432137679360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10325435584512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76548644 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10195186569297 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89393824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1459671167479808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚8235.697425197 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89403801 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10210419799552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16905608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7842810435681603e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20793363 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165942468 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333418824652800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333418719560704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10194846019407 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1469866371088465 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165952447 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1477431779300864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37698700 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76548646 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323093389068288™5½åñ2×- _wandbruntime‚8236  _runtime‚8236çÓÆ'Ø€N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37702799 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10326457157120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76556278 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1470013892814171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1477580032957440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89412867 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76556278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16907353 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1467368617570304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165969145 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37702511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20795158 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10195857073129 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76556276 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10211773772800 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89402876 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333453167801856 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1470013535224809 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842810441985483e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1467368617570304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323126799214592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 _runtime‚8236.324649236 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89402876 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1467368600530944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20795443 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165969145 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10196197623131 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚794 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1477580391343104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165959152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323126817826816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚9649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872{†B/_Ù…N _step‚794 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1477580391343104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165959152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323126817826816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/glof¾t›¬"bal_step‚9649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37702799 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10326457157120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76556278 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1470013892814171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1477580032957440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89412867 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76556278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16907353 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1467368617570304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165969145 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37702511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20795158 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10195857073129 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76556276 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10211773772800 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89402876 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333453167801856 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1470013535224809 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842810441985483e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1467368617570304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323126799214592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 _runtime‚8236.324649236 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89402876 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1467368600530944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20795443 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165969145 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10196197623131 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169š·I©2Ú- _wandbruntime‚8236  _runtime‚8236 òœõ Û… %loss/total_loss‚7.872478933024245 epoch‚24 trainer/global_step‚9649 _runtime‚ 8236.32612937 _step‚795 3loss/FBt2/assignment_loss‚0.0005027512460711674 0loss/FRt1/assignment_loss‚1.5698772389669133 0loss/FRt2/assignment_loss‚0.8769137250222061 3loss/SRqqt1/assignment_loss‚0.48543712571457587 2loss/SRqqt2/assignment_loss‚0.5310138535267854 /loss/FRt1/detection_loss‚0.5304479485721044 1loss/SRqqt2/detection_loss‚0.6153572049319384 % _timestamp‚1.7842810442007432e+09 2loss/FBt1/assignment_loss‚0.009142744768440991 1loss/SRqqt1/detection_loss‚0.9901085865653318 /loss/FBt1/detection_loss‚0.3260192953540744 /loss/FBt2/detection_loss‚0.6646642890582616 /loss/FRt2/detection_loss‚1.2729941692975413²î¦ÜŠ /loss/FRt2/detection_loss‚1.2729941692975413 %loss/total_loss‚7.872478933024245 epoch‚24 trainer/global_step‚9649 _runtime‚ 8236.32612937 _step‚795 3loss/FBt2/assignment_loss‚0.0005027512460711674 0loss/FRt1/assignment_loss‚1.5698772389669133 0loss/FRt2/assignment_loss‚0.8769137250222061 3loss/SRqqt1/assignment_loss‚0.48543712571457587 2loss/SRqqt2/assignment_loss‚0.5310138535267854 /loss/FRt1/detection_loss‚0.5304479485721044 1loss/SRqqt2/detection_loss‚0.6153572049319384 % _timestamp‚1.7842810442007432e+09 2loss/FBt1/assignment_loss‚0.009142744768440991 1loss/SRqqt1/detection_loss‚0.9901085865653318 /loss/FBt1/detection_loss‚0.3260192953540744 /loss/FBt2/detection_loss‚0.6646642890582616›x÷EÝ‚(:’ áïçÒ‘Õ†¤ gpu.0.gpu‚31.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚134.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.968200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I%*ñÞ‚(:æ áïçÒ澃¥) disk./.usagePercent‚36.82818989278978 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚36501.3896484375 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 395.765625! disk.rbd1.out‚104.44287109375 disk.nvme0n1.in‚ 395.765625 disk.rbd0.in‚0 disk.sda.out‚ 318.21875 disk.sda2.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚36501.3896484375 disk.rbd0.out‚0 disk.dm-1.out‚ 318.21875 disk.rbd3.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚548.7666435241699 disk.sda.in‚0 disk.sda3.out‚ 318.21875 disk.dm-1.in‚0" disk.dm-0.out‚36501.3896484375 disk.nvme0c0n1.in‚ 395.765625 network.recv‚ 2.185975e+06 disk.drbd2165.out‚0 disk.sda2.out‚0# memory_percent‚97.2337007522583( proc.memory.availableMB‚ 453.23046875* proc.memory.percent‚17.156696319580078 cpu‚26.054412935105905 network.sent‚ 1.5328919e+07 disk.sda1.in‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2810.953125NÚ·›“ß‚(:ˆ ðïçÒ†ÙÓ¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚230.288' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.0576 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0» Föà‚(:ë ðïçÒÑÏߤ disk.drbd2165.in‚0! disk.rbd1.out‚104.56787109375! proc.memory.rssMB‚ 2810.953125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 318.4140625 disk.dm-1.in‚0$ memory_percent‚97.23522663116455 disk.dm-0.in‚ 395.765625 network.sent‚ 1.5392058e+07* disk./.usagePercent‚36.828196804993205 disk.nvme0c0n1.in‚ 395.765625 disk.sda2.out‚0* proc.memory.percent‚17.156696319580078 disk.sda3.in‚0 disk.rbd3.in‚0 network.recv‚ 2.191215e+06$ disk./.usageGB‚548.7667465209961 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚36503.986328125 disk.sda.in‚0 disk.sda3.out‚ 318.4140625 disk.dm-1.out‚ 318.4140625 disk.rbd3.out‚0 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 452.98046875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 395.765625 disk.rbd0.out‚0 disk.sda2.in‚0! disk.dm-0.out‚36503.986328125 cpu‚26.053775101612914$ disk.nvme0n1.out‚36503.986328125 disk.sda1.out‚0Ä–³ž2á- _wandbruntime‚8278  _runtime‚8278´S4ÇâÁ & lr-AdamW/pg1‚0.0002552920435514719 & lr-AdamW/pg2‚0.0002552920435514719 trainer/global_step‚9699 % _timestamp‚1.7842810867254975e+09 _runtime‚8278.851143574 _step‚796d—õ‹ÌãÆ & lr-AdamW/pg2‚0.0002552920435514719 trainer/global_step‚9699 % _timestamp‚1.7842810867254975e+09 _runtime‚8278.851143574 _step‚796 & lr-AdamW/pg1‚0.0002552920435514719œXA2ä- _wandbruntime‚8278  _runtime‚8278[­ª›ü'åöO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1484844526284288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166788499 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1466996701073408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1474583410982912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10261456640512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16992858 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324764808382464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1484844526284288 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10261115301376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10261115301376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16992861 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89858253 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1477242163579723 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1466996718112768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1477242521168973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10245803056205 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76930246 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76930246 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89858253 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20897500 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚8278.852630922 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76930244 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37890093 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚797 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1474583428022272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10376634093568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89848276é\æ›!æûO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1474583410982912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76930244 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37890093 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚797 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1474583428022272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10376634093568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89848276 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1484844526284288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166788499 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1466996701073408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1474583410982912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10261456640512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16992858 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324764808382464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1484844526284288 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10261115301376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10261115301376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16992861 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89858253 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1477242163579723 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1466996718112768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1477242521168973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ –¢§]f612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10245803056205 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76930246 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76930246 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89858253 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20897500 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚8278.852630922 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166805197 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚8279.486523022 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166805197 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10262469274624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1477389685305317 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76937878 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37893978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1467143211745280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335175906355712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10377751493120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1474730652012544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10246814110039 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1484992779940864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166795204 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10262127928320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166795204 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89867319 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚798 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10262127928320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14747306690519040â¸2 'é„N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166795204 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10262127928320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166795204 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89867319 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚798 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10262127928320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1474730669051904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37894226 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20899620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324798255753216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10246473560037 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89867319 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76937878 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1477390042894679 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16994603 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20899375 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚9699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89857328 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1474730669051904 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89857328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1467143228784640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1474730652012544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16994606 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166805197 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚8279.486523022 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166805197 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10262469274624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1477389685305317 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76937878 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37893978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1467143211745280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335175906355712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10377751493120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1474730652012544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10246814110039 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1484992779940864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ž~‡­g2ê- _wandbruntime‚8279  _runtime‚8279ÕY¾Úëÿ 0loss/SRqqt2/detection_loss‚0.624001971084322 epoch‚24 _runtime‚8279.488433018 .loss/FRt1/assignment_loss‚1.56031911808761 1loss/SRqqt2/assignment_loss‚0.469429553873259 1loss/FBt1/assignment_loss‚0.06447676804848082 /loss/FRt1/detection_loss‚0.5346745351071189 1loss/SRqqt1/detection_loss‚0.9841789459018467 /loss/FBt2/detection_loss‚0.4691352996115501 _step‚799 0loss/FRt2/assignment_loss‚0.8979548763497212 2loss/SRqqt1/assignment_loss‚0.4663838544554353 1loss/FBt2/assignment_loss‚0.01749236981060256 /loss/FRt2/detection_loss‚1.2095792010785433 0loss/FBt1/detection_loss‚0.33214577721209143 %loss/total_loss‚7.629772270620581 % _timestamp‚1.7842810873629394e+09 trainer/global_step‚9699hÁa` ì„ .loss/FRt1/assignment_loss‚1.56031911808761 1loss/SRqqt2/assignment_loss‚0.469429553873259 1loss/FBt1/assignment_loss‚0.06447676804848082 /loss/FRt1/detection_loss‚0.5346745351071189 1loss/SRqqt1/detection_loss‚0.9841789459018467 /loss/FBt2/detection_loss‚0.4691352996115501 _step‚799 0loss/FRt2/assignment_loss‚0.8979548763497212 2loss/SRqqt1/assignment_loss‚0.4663838544554353 1loss/FBt2/assignment_loss‚0.01749236981060256 /loss/FRt2/detection_loss‚1.2095792010785433 0loss/FBt1/detection_loss‚0.33214577721209143 %loss/total_loss‚7.629772270620581 % _timestamp‚1.7842810873629394e+09 trainer/global_step‚9699 0loss/SRqqt2/detection_loss‚0.624001971084322 epoch‚24 _runtime‚8279.488433018Ÿ” ·‘í‚(:† ÿïçÒîüû¢ gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.231' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_â{òî‚(:ç ÿïçÒõ ø¤$ disk./.usageGB‚548.7666320800781 disk.dm-1.out‚ 318.68359375 disk.sda3.in‚0* proc.memory.percent‚17.156696319580078 network.recv‚ 2.194415e+06 disk.rbd3.in‚0 disk.nvme0n1.in‚ 396.59375$ disk.nvme0n1.out‚36513.779296875 disk.nvme0c0n1.in‚ 396.59375) disk./.usagePercent‚36.82818912476718 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 318.68359375 disk.dm-1.in‚0! disk.rbd1.out‚104.56787109375 disk.sda.in‚0 disk.sda3.out‚ 318.68359375 disk.rbd3.out‚0 disk.dm-0.in‚ 396.59375! disk.dm-0.out‚36513.779296875 disk.drbd2165.in‚0 disk.sda2.in‚0$ memory_percent‚97.23362922668457 disk.rbd1.in‚0.0625 disk.sda1.in‚0& disk.nvme0c0n1.out‚36513.779296875' proc.memory.availableMB‚ 453.2421875 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚26.0531965098536 disk.sda1.out‚0! proc.memory.rssMB‚ 2810.953125 proc.cpu.threads‚17 network.sent‚ 1.5396181e+07àŠÀï‚(:’ ŽðçÒñøУ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚254.229' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.84580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ž$`ûð‚(:ð ŽðçÒ®·¡¥* proc.memory.percent‚17.156696319580078 cpu‚26.05255925789025) disk./.usagePercent‚36.82818502864663$ disk./.usageGB‚548.7665710449219& disk.nvme0n1.out‚36521.21435546875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 397.32421875 disk.nvme0n1.in‚ 397.32421875 disk.rbd0.in‚0 disk.sda.out‚ 318.890625 disk.sda3.in‚0! disk.rbd1.out‚104.63037109375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚97.23367691040039& proc.memory.availableMB‚ 453.234375 disk.sda3.out‚ 318.890625 disk.rbd1.in‚0.0625 network.recv‚ 2.199541e+06 disk.rbd3.out‚0 disk.sda.in‚0! proc.memory.rssMB‚ 2810.953125 network.sent‚ 1.5459185e+07 disk.dm-1.out‚ 318.890625# disk.dm-0.out‚36521.21435546875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 397.32421875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚36521.21435546875r çñ‚(:’ ðçÒ˺ ¢ gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.589999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èäœýò‚(:ò ðçÒ…™ð¥ disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 cpu‚26.051982190818503 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.out‚ 320.921875 disk.drbd2165.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚36536.34033203125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 398.15234375( disk.nvme0c0n1.out‚36536.34033203125 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0# memory_percent‚97.2337007522583$ disk./.usageGB‚548.7617607116699 disk.drbd2165.in‚0 disk.sda2.out‚0 network.sent‚ 1.5464213e+07 network.recv‚ 2.202938e+06) disk./.usagePercent‚36.82786220314595 disk.nvme0n1.in‚ 398.15234375 disk.dm-0.in‚ 398.15234375 disk.sda.out‚ 320.921875 disk.sda2.in‚0 disk.dm-1.out‚ 320.921875! disk.rbd1.out‚104.63037109375( proc.memory.availableMB‚ 453.23046875! proc.memory.rssMB‚ 2810.953125* proc.memory.percent‚17.156696319580078& disk.nvme0n1.out‚36536.34033203125¹µ g2ó- _wandbruntime‚8319  _runtime‚8319‚‘ Éôà _runtime‚8319.183120954 _step‚800 ' lr-AdamW/pg1‚0.00025340351397030216 ' lr-AdamW/pg2‚0.00025340351397030216 trainer/global_step‚9749 % _timestamp‚1.7842811270575252e+09X‹¶ÎõÈ ' lr-AdamW/pg1‚0.00025340351397030216 ' lr-AdamW/pg2‚0.00025340351397030216 trainer/global_step‚9749 % _timestamp‚1.7842811270575252e+09 _runtime‚8319.183120954 _step‚800 %,‹¸2ö- _wandbruntime‚8319  _runtime‚8319ɸê÷öO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10427922966016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1492257273267712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77311846 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1474322251706368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚801 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1492257631646208 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1481945479503872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1481945462464512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167624551 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 _runtime‚8319.184493534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10427840680448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21000891 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1474322234667008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1481945462464512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90302728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17080111 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77311846 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38081275 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1484618671249481 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10296419543113 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77311844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167614572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90312705 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10311810803200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1492257273267712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90312705 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326436246308864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10312152142336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10311810803200 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1484618313660231 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17080108‘¿.Û(øûO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1492257273267712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77311846 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1474322251706368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚801 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1492257631646208 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1481945479503872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1481945462464512 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167624551 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 _runtime‚8319.184493534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10427840680448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21000891 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1474322234667008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1481945462464512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90302728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17080111 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77311846 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38081275 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1484618671249481 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10296419543113 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77311844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167614572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90312705 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10311810803200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1492257273267712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90312705 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326436246308864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10312152142336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10311810803200 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1484618313660231 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17080108 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10427922966016¡À€2ù- _wandbruntime‚8319  _runtime‚8319k¬'úÿM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21003231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326469693679616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10428856894464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38084821 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90311780 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17081853 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10312823430144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1492405526924288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21002968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167641249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1482092703494144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167631256 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1484765835385825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1482092720533504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17081856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1474468762378240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚ 8319.79527786 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77319476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167631256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77319478 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90311780 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚802 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10313164776448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167641249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991ñ8™»“û„N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326469693679616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10428856894464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38084821 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90311780 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17081853 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10312823430144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1492405526924288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21002968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167641249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1482092703494144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167631256 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1484765835385825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1482092720533504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17081856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1474468762378240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _runtime‚ 8319.79527786 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77319476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167631256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77319478 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90311780 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚802 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10313164776448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167641249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21003231¢\­Ü2ü- _wandbruntime‚8319  _runtime‚83190xÄ\ý‚ 1loss/SRqqt2/assignment_loss‚0.499038298688287 2loss/FBt1/assignment_loss‚0.021511562312512347 0loss/SRqqt1/detection_loss‚0.893111278700587 0loss/FBt1/detection_loss‚0.40862857587080426 trainer/global_step‚9749 /loss/FRt1/detection_loss‚0.5196260387222013 /loss/FRt2/detection_loss‚1.2421288393041239 $ _timestamp‚1.784281127672925e+09 _runtime‚8319.798274314 3loss/FBt2/assignment_loss‚0.0003506952531389107 1loss/SRqqt2/detection_loss‚0.6594928935415422 /loss/FBt2/detection_loss‚0.6357374634826556 %loss/total_loss‚7.707034227081442 _step‚803 /loss/FRt1/assignment_loss‚1.567446942473826 /loss/FRt2/assignment_loss‚0.833623346316722 3loss/SRqqt1/assignment_loss‚0.42633829241504123 epoch‚24£+.² þ‡ 1loss/SRqqt2/assignment_loss‚0.499038298688287 2loss/FBt1/assignment_loss‚0.021511562312512347 0loss/SRqqt1/detection_loss‚0.893111278700587 0loss/FBt1/detection_loss‚0.40862857587080426 trainer/global_step‚9749 /loss/FRt1/detection_loss‚0.5196260387222013 /loss/FRt2/detection_loss‚1.2421288393041239 $ _timestamp‚1.784281127672925e+09 _runtime‚8319.798274314 3loss/FBt2/assignment_loss‚0.0003506952531389107 1loss/SRqqt2/detection_loss‚0.6594928935415422 /loss/FBt2/detection_loss‚0.6357374634826556 %loss/total_loss‚7.707034227081442 _step‚803 /loss/FRt1/assignment_loss‚1.567446942473826 /loss/FRt2/assignment_loss‚0.833623346316722 3loss/SRqqt1/assignment_loss‚0.42633829241504123 epoch‚24£eu-[›ÿ‚(: ¬ðçÒª„¬¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.921' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.38419999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç}3€ ‚(:ö ¬ðçÒ…Ô¡¦( disk.nvme0c0n1.out‚36544.56591796875 disk.dm-1.in‚0 network.sent‚ 1.5469322e+07 disk.rbd3.in‚0 disk.nvme0n1.in‚ 398.8671875& disk.nvme0n1.out‚36544.56591796875 disk.rbd0.out‚0 disk.sda.out‚ 321.76953125$ memory_percent‚97.23217487335205( proc.memory.availableMB‚ 453.48046875 disk.sda3.out‚ 321.76953125! proc.memory.rssMB‚ 2810.953125* proc.memory.percent‚17.156696319580078$ disk./.usageGB‚548.7622375488281 disk.dm-0.in‚ 398.8671875 disk.sda1.out‚0 disk.sda3.in‚0 cpu‚26.051408637900302 network.recv‚ 2.206605e+06) disk./.usagePercent‚36.82789420408773 disk.rbd3.out‚0# disk.dm-0.out‚36544.56591796875 disk.sda.in‚0 proc.cpu.threads‚17! disk.rbd1.out‚104.69287109375 disk.drbd2165.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 398.8671875 disk.sda2.in‚0 disk.dm-1.out‚ 321.76953125 disk.sda1.in‚0 disk.rbd1.in‚0.0625 disk.sda2.out‚0 disk.drbd2165.in‚0 Kø ‚(:’ »ðçÒ˜¯Ÿ¤ gpu.0.gpu‚80.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gu K‚ ‚(:÷ »ðçÒ³ÈÁ¥* disk./.usagePercent‚36.827843514595955 disk.dm-0.in‚ 398.8671875# disk.dm-0.out‚36550.96337890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 322.14453125 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 322.14453125 disk.sda3.out‚ 322.14453125 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚36550.96337890625! disk.rbd1.out‚104.69287109375$ memory_percent‚97.23031520843506 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 network.recv‚ 2.212055e+06! disk.nvme0c0n1.in‚ 398.8671875! proc.memory.rssMB‚ 2810.953125$ disk./.usageGB‚548.7614822387695 disk.dm-1.in‚0( proc.memory.availableMB‚ 453.78515625 proc.cpu.threads‚17 network.sent‚ 1.5532395e+07 disk.nvme0n1.in‚ 398.8671875& disk.nvme0n1.out‚36550.96337890625 cpu‚26.050814747433897 disk.rbd3.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.156696319580078<U¿žƒ ‚(:“ ÊðçÒùÁ¯¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.889' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.777800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÁû„ ‚(:ð ÊðçÒ³¡Û¤* disk./.usagePercent‚36.827861947138416 disk.sda1.out‚0 disk.dm-1.out‚ 322.3359375 disk.rbd3.out‚0 disk.dm-0.in‚ 399.6953125( disk.nvme0c0n1.out‚36558.01318359375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0! disk.rbd1.out‚104.75537109375! disk.nvme0c0n1.in‚ 399.6953125 disk.sda.out‚ 322.3359375! proc.memory.rssMB‚ 2810.953125 cpu‚26.05022318417296 proc.cpu.threads‚17$ disk./.usageGB‚548.7617568969727 disk.nvme0n1.in‚ 399.6953125 disk.rbd0.in‚0* proc.memory.percent‚17.156696319580078 network.recv‚ 2.215463e+06& disk.nvme0n1.out‚36558.01318359375 disk.sda2.in‚0 disk.sda3.out‚ 322.3359375 network.sent‚ 1.5537453e+07 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0% proc.memory.availableMB‚ 453.28125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚36558.01318359375$ memory_percent‚97.23339080810547¼-ÔÛ2… - _wandbruntime‚8358  _runtime‚8358:_çjɆ à trainer/global_step‚9799 % _timestamp‚1.7842811667773001e+09 _runtime‚8358.902897627 _step‚804 ' lr-AdamW/pg1‚0.00025151388805301706 ' lr-AdamW/pg2‚0.00025151388805301706?jL…· È _runtime‚8358.902897627 _step‚804 ' lr-AdamW/pg1‚0.00025151388805301706 ' lr-AdamW/pg2‚0.00025151388805301706 trainer/global_step‚9799 % _timestamp‚1.7842811667773001e+09¤>Ù˜¥2ˆ - _wandbruntime‚8358  _runtime‚8358´ÉÝÓü'‰ öO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1481647785299968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1489307513946112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90757180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚805 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1499670020251136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90767157 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1491994463740739 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1499670020251136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10362506305024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1499670378629632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168450624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77693446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77693444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10362847644160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38271644 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚338586829537280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1481647768260608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1499670378629632 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17167361 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38271362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21104283 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10362847644160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚8358.904115486¾–XQ(Š ûO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1489307513946112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90757180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _step‚805 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1499670020251136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90767157 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1491994463740739 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1499670020251136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10362506305024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1499670378629632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168450624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77693446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77693444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10362847644160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38271644 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚338586829537280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1481647768260608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1499670378629632 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17167361 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38271362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21104283 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10362847644160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚8358.904115486 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1481647785299968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10348047083855 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17169106 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10347706533853 UHDevic;x?eStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168477301 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21106383 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚338621279630848 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10363860278272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1499818632293376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17169103 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328141112993792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10363518931968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77701078 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10480148024832 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842811673896627e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38275224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1492142343055695 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77701076 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168467308 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 8359.51545818 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17169103 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328141112993792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10363518931968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77701078 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10480148024832 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842811673896627e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38275224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1492142343055695 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77701076 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168467308 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 8359.51545818 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10348047083855 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17169106 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10347706533853 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168477301 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21106383 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚338621279630848 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664¦<Ê"2Ž - _wandbruntime‚8359  _runtime‚8359,Æ(.  ƒ /loss/FBt2/detection_loss‚0.7961813991941098 epoch‚24 2loss/SRqqt1/assignment_loss‚0.4074979273881763 1loss/SRqqt2/detection_loss‚0.6149921055814418 0loss/FBt1/detection_loss‚0.41503570968011666 trainer/global_step‚9799 $ _timestamp‚1.784281167392923e+09 _step‚807 0loss/FRt2/assignment_loss‚0.8628518809010249 2loss/SRqqt2/assignment_loss‚0.5443429599972505 %loss/total_loss‚7.937806956135471 0loss/FRt1/assignment_loss‚1.6293071423811618 2loss/FBt1/assignment_loss‚0.008833498920744864 3loss/FBt2/assignment_loss‚0.0014901088935354147 /loss/FRt1/detection_loss‚0.4998641340205137 1loss/SRqqt1/detection_loss‚0.8609364954996854 _runtime‚8359.518270929 -loss/FRt2/detection_loss‚1.29647359367771dA!ÿ ˆ %loss/total_loss‚7.937806956135471 0loss/FRt1/assignment_loss‚1.6293071423811618 2loss/FBt1/assignment_loss‚0.008833498920744864 3loss/FBt2/assignment_loss‚0.0014901088935354147 /loss/FRt1/detection_loss‚0.4998641340205137 1loss/SRqqt1/detection_loss‚0.8609364954996854 _runtime‚8359.518270929 -loss/FRt2/detection_loss‚1.29647359367771 /loss/FBt2/detection_loss‚0.7961813991941098 epoch‚24 2loss/SRqqt1/assignment_loss‚0.4074979273881763 1loss/SRqqt2/detection_loss‚0.6149921055814418 0loss/FBt1/detection_loss‚0.41503570968011666 trainer/global_step‚9799 $ _timestamp‚1.784281167392923e+09 _step‚807 0loss/FRt2/assignment_loss‚0.8628518809010249 2loss/SRqqt2/assignment_loss‚0.5443429599972505§iÆÕ’‘ ‚(:‡ ÙðçÒ¢ó£¥ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.201 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûvd’ ‚(:ù ÙðçÒÞ¸§ disk.nvme0n1.in‚ 400.41015625* disk./.usagePercent‚36.827537329585006 disk.rbd3.in‚0# disk.dm-0.out‚36566.76318359375 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 network.sent‚ 1.5542469e+07 disk.rbd0.in‚0* proc.memory.percent‚17.156696319580078 cpu‚26.04958133608663 disk.rbd3.out‚0 disk.dm-0.in‚ 400.41015625 disk.sda1.out‚0 disk.sda3.out‚ 322.51953125 disk.dm-1.out‚ 322.51953125! disk.rbd1.out‚104.75537109375( proc.memory.availableMB‚ 453.54296875 proc.cpu.threads‚17$ disk./.usageGB‚548.7569198608398& disk.nvme0n1.out‚36566.76318359375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚36566.76318359375 disk.sda.out‚ 322.51953125 disk.drbd2165.in‚0 disk.sda2.out‚0$ memory_percent‚97.23179340362549! proc.memory.rssMB‚ 2810.953125 network.recv‚ 2.219133e+06" disk.nvme0c0n1.in‚ 400.41015625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda.in‚0œ9…‘“ ‚(:† èðçҾ뾢 gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï¿Š1” ‚(:ö èðçÒçüž¤ network.sent‚ 1.5605739e+07 disk.rbd3.in‚0 disk.nvme0n1.in‚ 401.23828125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚36573.85009765625 disk.sda1.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚548.7572860717773 disk.sda3.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚104.84130859375* proc.memory.percent‚17.156696319580078) disk./.usagePercent‚36.82756190630829( disk.nvme0c0n1.out‚36573.85009765625 disk.sda.in‚0 disk.drbd2165.in‚0 cpu‚26.04900962429163 disk.dm-1.out‚ 322.83203125 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 401.23828125 disk.sda1.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚36573.85009765625 network.recv‚ 2.224306e+06" disk.nvme0c0n1.in‚ 401.23828125 disk.sda.out‚ 322.83203125 disk.sda3.out‚ 322.83203125$ memory_percent‚97.23291397094727& proc.memory.availableMB‚ 453.359375! proc.memory.rssMB‚ 2810.953125³ÕØ2• - _wandbruntime‚8398  _runtime‚8398' ±É– à ' lr-AdamW/pg2‚0.00024962328716675834 trainer/global_step‚9849 % _timestamp‚1.7842812067745354e+09 _runtime‚8398.900082391 _step‚808 ' lr-AdamW/pg1‚0.00024962328716675834¼VúΗ È ' lr-AdamW/pg1‚0.00024962328716675834 ' lr-AdamW/pg2‚0.00024962328716675834 trainer/global_step‚9849 % _timestamp‚1.7842812067745354e+09 _runtime‚8398.900082391 _step‚808¨‚G™¦2˜ - _wandbruntime‚8398  _runtime‚8398ÿÂÍé'™ õO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _runtime‚8398.901358129 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91221609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1507083125613056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1496669565427712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21207754 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340309389533696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329779140773888 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1499370971410497 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10413543145984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10413201806848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1499370613821247 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10397311967039 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91221609 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1496669565427712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91211632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78075044 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784281206775538e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10530267372032 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1488973301854208 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1507083125613056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340309498820096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1496669582467072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38462362 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91211632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78075046 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1507082767234560 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78075046 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10530358046208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17254611 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10413543145984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17254608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329779122161664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1496669582467072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21208022 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38462633 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.R}8peak‚ 23702949376f¥2(š úO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78075044 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10530267372032 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1488973301854208 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1507083125613056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340309498820096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1496669582467072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38462362 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91211632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78075046 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1507082767234560 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78075046 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10530358046208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17254611 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10413543145984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17254608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329779122161664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1496669582467072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21208022 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38462633 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _runtime‚8398.901358129 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91221609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1507083125613056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1496669565427712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21207754 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340309389533696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329779140773888 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1499370971410497 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9849 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10413543145984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10413201806848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1499370613821247 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10397311967039 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91221609 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1496669565427712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91211632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78075044 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784281206775538e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91220684 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340343835947520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10414214433792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91230675 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7842812073879442e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1496816823496704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21209802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10414555780096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10531285027328 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1489119812526080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10531379888640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91220684 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78082676 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10398663570763 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1507231020891136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340343949421056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚9849 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38466155 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38466426 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1507231379276800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313353 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17256353 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313353 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38466155 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38466426 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1507231379276800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313353 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17256353 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313353 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91220684 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340343835947520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10414214433792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91230675 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7842812073879442e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1496816823496704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21209802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10414555780096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10531285027328 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1489119812526080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10531379888640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91220684 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78082676 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10398663570763 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1507231020891136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340343949421056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚9849 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968ªº¢Ë!2ž - _wandbruntime‚8399  _runtime‚8399®»áŸ ˆ /loss/FRt2/detection_loss‚1.2064074745157207 %loss/total_loss‚7.724840661582487 epoch‚24 % _timestamp‚1.7842812073902264e+09 2loss/SRqqt1/assignment_loss‚0.5036828785668429 /loss/FRt1/detection_loss‚0.5504263945615139 1loss/SRqqt2/detection_loss‚0.6175752150543883 _runtime‚8399.515556337 _step‚811 0loss/FRt2/assignment_loss‚0.8467250471511788 4loss/FBt2/assignment_loss‚0.00017798943578682265 /loss/FBt2/detection_loss‚0.5992779285616933 trainer/global_step‚9849 0loss/FRt1/assignment_loss‚1.5807354922151247 1loss/SRqqt1/detection_loss‚0.8887676046284327 0loss/FBt1/detection_loss‚0.45491438424068964 2loss/SRqqt2/assignment_loss‚0.4702449679915835 3loss/FBt1/assignment_loss‚0.0059052846595316795˜š   0loss/FRt2/assignment_loss‚0.8467250471511788 4loss/FBt2/assignment_loss‚0.00017798943578682265 /loss/FBt2/detection_loss‚0.5992779285616933 trainer/global_step‚9849 0loss/FRt1/assignment_loss‚1.5807354922151247 1loss/SRqqt1/detection_loss‚0.8887676046284327 0loss/FBt1/detection_loss‚0.45491438424068964 2loss/SRqqt2/assignment_loss‚0.4702449679915835 3loss/FBt1/assignment_loss‚0.0059052846595316795 /loss/FRt2/detection_loss‚1.2064074745157207 %loss/total_loss‚7.724840661582487 epoch‚24 % _timestamp‚1.7842812073902264e+09 2loss/SRqqt1/assignment_loss‚0.5036828785668429 /loss/FRt1/detection_loss‚0.5504263945615139 1loss/SRqqt2/detection_loss‚0.6175752150543883 _runtime‚8399.515556337 _step‚811«'í5œ¡ ‚(:‘ ÷ðçÒ·Ò¹¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jPyv¢ ‚(:è ÷ðçÒ€«Ó¤ disk.sda.out‚ 323.1328125 disk.sda2.in‚0! proc.memory.rssMB‚ 2810.953125* proc.memory.percent‚17.156696319580078 disk.sda3.in‚0! disk.rbd1.out‚104.85302734375 disk.drbd2165.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚36582.365234375 disk.sda.in‚0 disk.sda3.out‚ 323.1328125 disk.dm-1.out‚ 323.1328125~Y›} disk.rbd1.in‚0.0625$ disk.nvme0n1.out‚36582.365234375 disk.sda1.in‚0 disk.rbd3.out‚0$ memory_percent‚97.23515510559082) disk./.usagePercent‚36.82758929911445! disk.dm-0.out‚36582.365234375 disk.nvme0n1.in‚ 401.953125 cpu‚26.04841675349737 network.sent‚ 1.5610746e+07$ disk./.usageGB‚548.7576942443848 disk.sda2.out‚0 disk.dm-0.in‚ 401.953125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 401.953125 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 452.9921875 proc.cpu.threads‚17 network.recv‚ 2.227822e+06 Áx.‘£ ‚(:† †ñçÒ——¢ gpu.0.gpu‚7.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æle¾÷¤ ‚(:ì †ñçÒÔŽ‚¦ disk.rbd0.in‚0$ memory_percent‚97.23482131958008 proc.cpu.threads‚17 disk.sda2.in‚0 network.recv‚ 2.23341e+06 disk.nvme0n1.in‚ 401.953125 disk.drbd2165.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.156696319580078 cpu‚26.04784934386959 disk.sda3.in‚0 disk.sda3.out‚ 323.3359375 disk.nvme0c0n1.in‚ 401.953125 disk.sda.in‚0# disk.dm-0.out‚36586.43701171875& disk.nvme0n1.out‚36586.43701171875 disk.sda.out‚ 323.3359375 disk.sda2.out‚0& proc.memory.availableMB‚ 453.046875 network.sent‚ 1.567413e+07 disk.rbd3.out‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2810.953125 disk.dm-1.in‚0 disk.dm-1.out‚ 323.3359375 disk.dm-0.in‚ 401.953125 disk.drbd2165.in‚0* disk./.usagePercent‚36.827480495912404 disk.rbd1.in‚0.0625( disk.nvme0c0n1.out‚36586.43701171875 disk.sda1.out‚0$ disk./.usageGB‚548.7560729980469! disk.rbd1.out‚105.03857421875 disk.rbd3.in‚0cÚ´¥ ‚(:’ •ñçÒ®–¹¢ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦»ÎÙô¦ ‚(:é •ñçÒ•ÃŤ$ memory_percent‚97.23570346832275 disk.rbd1.in‚0.0625" disk.dm-0.out‚36596.7001953125 disk.nvme0n1.in‚ 402.78125 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 402.78125' disk.nvme0c0n1.out‚36596.7001953125* proc.memory.percent‚17.156696319580078! disk.rbd1.out‚105.03857421875 disk.dm-0.in‚ 402.78125 disk.drbd2165.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 452.90234375! proc.memory.rssMB‚ 2810.953125 disk.sda.out‚ 323.5390625 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 323.5390625 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚26.04724313401282 network.recv‚ 2.236906e+06* disk./.usagePercent‚36.827748535800744# disk./.usageGB‚548.760066986084 disk.dm-1.in‚0 disk.dm-1.out‚ 323.5390625 network.sent‚ 1.5679065e+07 disk.sda.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚36596.7001953125åAê42§ - _wandbruntime‚8438  _runtime‚8438ÛªÚŨ ¿ % lr-AdamW/pg2‚0.000247731832741288 trainer/global_step‚9899 % _timestamp‚1.7842812467148006e+09 _runtime‚8438.840402964 _step‚812 % lr-AdamW/pg1‚0.000247731832741288ü3•ìÊ© Ä trainer/global_step‚9899 % _timestamp‚1.7842812467148006e+09 _runtime‚8438.840402964 _step‚812 % lr-AdamW/pg1‚0.000247731832741288 % lr-AdamW/pg2‚0.000247731832741288¬gµ¦J2ª - _wandbruntime‚8438  _runtime‚8438íem‘ü'« öO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170132707 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331450560088064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10448269003837 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10447928453947 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1504031616909312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21311522 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78456644 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91666084 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1504031633948672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91676061 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10581449088000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17341861 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91676061 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21311788 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38653649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514495514217984 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7842812467158706e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170122728 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1504031633948672 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1514495514217984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚9899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78456644 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91666084 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1496298835447808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10464238647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1496298852487168 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514495872596480 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1514495872596480 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1506746763901755 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚813 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78456646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17341858 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342032120559616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331450578700288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10581541859328 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1504031616909312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38653380 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1506747121491005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ¤ý(¬ ûO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1506746763901755 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚813 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78456646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17341858 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342032120559616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331450578700288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10581541859328 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1504031616909312 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38653380 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1506747121491005 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170132707 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331450560088064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10448269003837 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10447928453947 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1504031616909312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21311522 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78456644 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91666084 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1504031633948672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91676061 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10581449088000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17341861 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91676061 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21311788 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38653649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514495514217984 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7842812467158706e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170122728 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1504031633948672 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1514495514217984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚9899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78456644 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91666084 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1496298835447808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10464238647808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1496298852487168 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514495872596480 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1514495872596480 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169­L¬B2­ - _wandbruntime‚8439  _runtime‚8439ægÓ¼í® €N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170149405 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38657116 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚814 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170149405 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78464276 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 % _timestamp‚1.7842812473274674e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10582563534336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91685127 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1504178857938944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10465251281920 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17343606 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91675136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78464278 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1506894643216711 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10449280057671 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1514644126260224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10448939507669 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚8439.453388011 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1504178874978304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78464278 cODeviceStatsMonitor.o¤R³nn_train_batch_end/inactive_split_bytes.large_pool.allocated‚331484007458816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1506894285627349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170139412 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1504178874978304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10582479158784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚9899 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1514644126260224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚342066570993152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331483988846592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21313776^kºõ '¯ …N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170149405 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78464276 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 % _timestamp‚1.7842812473274674e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10582563534336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91685127 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1504178857938944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10465251281920 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17343606 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91675136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78464278 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1506894643216711 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10449280057671 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1514644126260224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10448939507669 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚8439.453388011 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1504178874978304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78464278 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331484007458816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1506894285627349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170139412 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1504178874978304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10582479158784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚9899 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1514644126260224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚342066570993152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331483988846592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21313776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170149405 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38657116 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚814 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2®³W_¤2° - _wandbruntime‚8439  _runtime‚8439YV¨± ‚ 0loss/FRt2/assignment_loss‚0.9278236585236638 3loss/FBt2/assignment_loss‚0.0006473921076395328 .loss/FRt2/detection_loss‚1.244697060234245 0loss/FBt1/detection_loss‚0.39457767930220167 $loss/total_loss‚8.02731839105535 trainer/global_step‚9899 _runtime‚8439.456106952 _step‚815 3loss/SRqqt1/assignment_loss‚0.45102194674846907 0loss/SRqqt1/detection_loss‚1.033296672511446 1loss/SRqqt2/assignment_loss‚0.451798045907334 1loss/SRqqt2/detection_loss‚0.6201364291584802 /loss/FBt2/detection_loss‚0.7349279693277951 epoch‚24 0loss/FRt1/assignment_loss‚1.6270314268808108 2loss/FBt1/assignment_loss‚0.013222736969098139 /loss/FRt1/detection_loss‚0.5281373733841669 $ _timestamp‚1.784281247330767e+09_ଠ² ‡ 1loss/SRqqt2/detection_loss‚0.6201364291584802 /loss/FBt2/detection_loss‚0.7349279693277951 epoch‚24 0loss/FRt1/assignment_loss‚1.6270314268808108 2loss/FBt1/assignment_loss‚0.013222736969098139 /loss/FRt1/detection_loss‚0.5281373733841669 $ _timestamp‚1.784281247330767e+09 0loss/FRt2/assignment_loss‚0.9278236585236638 3loss/FBt2/assignment_loss‚0.0006473921076395328 .loss/FRt2/detection_loss‚1.244697060234245 0loss/FBt1/detection_loss‚0.39457767930220167 $loss/total_loss‚8.02731839105535 trainer/global_step‚9899 _runtime‚8439.456106952 _step‚815 3loss/SRqqt1/assignment_loss‚0.45102194674846907 0loss/SRqqt1/detection_loss‚1.033296672511446 1loss/SRqqt2/assignment_loss‚0.451798045907334¯Ùù³ ‚(:ø ¤ñçÒ†½°¥! disk.rbd1.out‚105.14794921875 disk.rbd0.in‚0$ memory_percent‚97.23398685455322( proc.memory.availableMB‚ 453.18359375 cpu‚26.04662435810849 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 network.sent‚ 1.5684057e+07 disk.rbd3.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.156696319580078$ disk./.usageGB‚548.7620086669922 disk.dm-0.in‚ 403.49609375" disk.nvme0c0n1.in‚ 403.49609375 disk.sda3.in‚0 disk.dm-1.out‚ 324.00390625 disk.nvme0n1.in‚ 403.49609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚ 324.00390625# disk.dm-0.out‚36604.41357421875( disk.nvme0c0n1.out‚36604.41357421875 disk.sda.out‚ 324.00390625! proc.memory.rssMB‚ 2810.953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0& disk.nvme0n1.out‚36604.41357421875 network.recv‚ 2.240389e+06) disk./.usagePercent‚36.82787884363567uôÅs´ ‚(:’ ¤ñçÒ°ØÃ¥ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿¡ìˆÿµ ‚(:ô ³ñçÒÐñž¤# disk.dm-0.out‚36613.52783203125 network.recv‚ 2.245841e+06$ disk./.usageGB‚548.7625350952148 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 324.1953125 disk.rbd1.in‚0.0625! proc.memory.rssMB‚ 2810.953125 network.sent‚ 1.5747108e+07 disk.nvme0n1.in‚ 404.32421875" disk.nvme0c0n1.in‚ 404.32421875 disk.sda.in‚0 disk.sda.out‚ 324.1953125 disk.sda3.in‚0' proc.memory.availableMB‚ 452.7421875* proc.memory.percent‚17.156696319580078 proc.cpu.threads‚17$ memory_percent‚97.23668098449707 disk.rbd3.out‚0 disk.dm-0.in‚ 404.32421875 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 324.1953125 disk.rbd3.in‚0( disk./.usagePercent‚36.8279141726754& disk.nvme0n1.out‚36613.52783203125 disk.sda1.in‚0 disk.sda2.out‚0! disk.rbd1.out‚105.14794921875 cpu‚26.046079106562377 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚36613.52783203125À`,“¶ ‚(:ˆ ³ñçÒÈ„¼ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚239.197' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.8394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÈ ’· ‚(:‡ ÂñçÒÐáÅ¢ gpu.0.gpu‚70.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚249.925' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.985 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÞÌü¸ ‚(:ñ ÂñçÒþŒø¤* disk./.usagePercent‚36.827908284502115 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 324.41015625$ memory_percent‚97.23672866821289 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.sda1.out‚0 cpu‚26.04552405459709 disk.dm-0.in‚ 404.32421875 disk.sda.out‚ 324.41015625 disk.sda2.in‚0! proc.memory.rssMB‚ 2810.953125$ disk.nvme0n1.out‚36616.431640625 disk.drbd2165.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 404.32421875* proc.memory.percent‚17.156696319580078 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 324.41015625! disk.rbd1.out‚105.21826171875 disk.nvme0n1.in‚ 404.32421875 disk.sda1.in‚0& proc.memory.availableMB‚ 452.734375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚36616.431640625 network.recv‚ 2.249213e+06$ disk./.usageGB‚548.7624473571777 disk.drbd2165.out‚0 network.sent‚ 1.5752127e+07 disk.rbd3.in‚0! disk.dm-0.out‚36616.431640625 disk.sda.in‚0Ñßî2¹ - _wandbruntime‚8478  _runtime‚84787z¿ Ǻ Á & lr-AdamW/pg2‚0.0002458396462611894 trainer/global_step‚9949 % _timestamp‚1.7842812864109938e+09 _runtime‚8478.536651542 _step‚816 & lr-AdamW/pg1‚0.0002458396462611894µÍØrÌ» Æ & lr-AdamW/pg1‚0.0002458396462611894 & lr-AdamW/pg2‚0.0002458396462611894 trainer/global_step‚9949 % _timestamp‚1.7842812864109938e+09 _runtime‚8478.536651542 _step‚816°MF¾12¼ - _wandbruntime‚8478  _runtime‚8478ù5’:û'½ õO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10514592810496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92130513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1511393668390912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92120536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1503624386080768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10498544940855 % _timestamp‚1.7842812864120984e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10498885490745 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚8478.538080139 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21415215 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10632738870784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1521908261201408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10632819059200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92120536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170958780 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78838246 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78838244 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1521908619579904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚817 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1511393685430272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21415464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78838244 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78838246 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92130513 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17429111 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1511393685430272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10514934149632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343754835685888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333121998014464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10514592810496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1521908261201408àÅuSl¾ úO ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1511393668390912 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17429108 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343754736885248 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170958780 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78838246 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78838244 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1521908619579904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚817 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1511393685430272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21415464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78838244 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78838246 TGDeviceStats˜bûx”Monitor.on_train_batch_start/allocation.small_pool.allocated‚92130513 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17429111 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1511393685430272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10514934149632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343754835685888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333121998014464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10514592810496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1521908261201408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10514592810496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92130513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1511393668390912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92120536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1503624386080768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10498544940855 % _timestamp‚1.7842812864120984e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10498885490745 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚8478.538080139 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21415215 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10632738870784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1521908261201408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10632819059200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92120536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1511540909420544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21417301 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343789288887808 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1514270793297219 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1511540926459904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78845876 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38848427 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333155426772992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10515605437440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78845878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170985457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343789179608576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10515946783744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92139579 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92129588 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842812870242836e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92129588 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1514270435707857 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170975464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17430853 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92139579 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1522056873243648 _step‚818 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10499555994577 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚8479.150075072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21417571 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1503770896752640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚9949 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10633752835584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1522056873243648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38848154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10499896544579 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1522056514857984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343789288887808 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1514270793297219 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1511540926459904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78845876 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38848427 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333155426772992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10515605437440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78845878 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170985457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343789179608576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10515946783744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92139579 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92129588 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842812870242836e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92129588 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1514270435707857 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170975464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17430853 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92139579 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1522056873243648 _step‚818 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10499555994577 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚8479.150075072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21417571 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1503770896752640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚9949 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10633752835584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1522056873243648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38848154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10499896544579 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1522056514857984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0²š×Öj2 - _wandbruntime‚8479  _runtime‚8479uot à ‡ 0loss/FBt2/detection_loss‚0.44023575101579937 3loss/SRqqt2/assignment_loss‚0.45432035332154713 /loss/FRt1/detection_loss‚0.5177766459748698 /loss/FRt2/detection_loss‚1.1240431674600053 1loss/SRqqt1/detection_loss‚0.7542672310655396 1loss/SRqqt2/detection_loss‚0.5299010727771254 /loss/FBt1/detection_loss‚0.3847110704104616 trainer/global_step‚9949 /loss/FRt1/assignment_loss‚1.484894824067015 epoch‚24 _runtime‚8479.153015694 0loss/FRt2/assignment_loss‚0.7910729442177148 3loss/FBt2/assignment_loss‚0.0055163757745076295 _step‚819 2loss/SRqqt1/assignment_loss‚0.3748898180062585 2loss/FBt1/assignment_loss‚0.020509830007319926 &loss/total_loss‚6.8821390840981636 % _timestamp‚1.7842812870276773e+09w“&GÄ Œ 0loss/FBt2/detection_loss‚0.44023575101579937 3loss/SRqqt2/assignment_loss‚0.45432035332154713 /loss/FRt1/detection_loss‚0.5177766459748698 /loss/FRt2/detection_loss‚1.1240431674600053 1loss/SRqqt1/detection_loss‚0.7542672310655396 1loss/SRqqt2/detection_loss‚0.5299010727771254 /loss/FBt1/detection_loss‚0.3847110704104616 trainer/global_step‚9949 /loss/FRt1/assignment_loss‚1.484894824067015 epoch‚24 _runtime‚8479.153015694 0loss/FRt2/assignment_loss‚0.7910729442177148 3loss/FBt2/assignment_loss‚0.0055163757745076295 _step‚819 2loss/SRqqt1/assignment_loss‚0.3748898180062585 2loss/FBt1/assignment_loss‚0.020509830007319926 &loss/total_loss‚6.8821390840981636 % _timestamp‚1.7842812870276773e+09³Õ4añ“Å ‚(:ˆ ÑñçÒø¿ý¢ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚255.341' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ÿ¿÷Æ ‚(:ì ÑñçÒÅ­‘¥! disk.nvme0c0n1.in‚ 405.0390625$ memory_percent‚97.23966121673584* proc.memory.percent‚17.156696319580078 disk.dm-0.in‚ 405.0390625" disk.dm-0.out‚36625.5771484375 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 405.0390625' disk.nvme0c0n1.out‚36625.5771484375 disk.sda2.out‚0 disk.dm-1.out‚ 324.640625 network.recv‚ 2.252855e+06 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2810.953125 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 452.25390625 cpu‚26.04491562103673) disk./.usagePercent‚36.82790521241171$ disk./.usageGB‚548.7624015808105! disk.rbd1.out‚105.21826171875 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 324.640625 disk.sda3.out‚ 324.640625 network.sent‚ 1.5757174e+07% disk.nvme0n1.out‚36625.57714843756È“çžÇ ‚(:“ àñçÒ ’æ£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…¼øÈ ‚(:í àñçÒþÝý¤ disk.sda.out‚ 325.078125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚36632.5673828125 disk.sda2.in‚0! proc.memory.rssMB‚ 2810.953125 proc.cpu.threads‚17 network.sent‚ 1.5820299e+07 disk.sda2.out‚0! disk.rbd1.out‚105.32763671875 cpu‚26.044340750684714 network.recv‚ 2.258011e+06 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚97.23827838897705 disk.nvme0n1.in‚ 405.8671875 disk.drbd2165.in‚0 disk.sda3.out‚ 325.078125 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 405.8671875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 325.078125" disk.dm-0.out‚36632.5673828125* proc.memory.percent‚17.156696319580078) disk./.usagePercent‚36.82791878081102 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 405.8671875( proc.memory.availableMB‚ 452.48046875$ disk./.usageGB‚548.7626037597656% disk.nvme0n1.out‚36632.5673828125—jE%2É - _wandbruntime‚8518  _runtime‚8518÷¨DXÉÊ Ã % _timestamp‚1.7842813264048944e+09 _runtime‚8518.530480361 _step‚820 ' lr-AdamW/pg1‚0.00024394684925806414 ' lr-AdamW/pg2‚0.00024394684925806414 trainer/global_step‚9999DШèÎË È ' lr-AdamW/pg1‚0.00024394684925806414 ' lr-AdamW/pg2‚0.00024394684925806414 trainer/global_step‚9999 % _timestamp‚1.7842813264048944e+09 _runtime‚8518.530480361 _step‚820´ óîú2Ì - _wandbruntime‚8518  _runtime‚8518(±^ØMÍ öO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10565629651456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10565629651456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10684062886400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10565288312320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21519282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1529321008184832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171804811 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚8518.535404914 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1529321366563328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79219846 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79219846 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1521499064062771 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92574988 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10549161427763 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9999 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345477603919360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1521499421652021 _step‚821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17516358 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10684149366272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10565288312320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345477498827264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1529321366563328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79219844 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17516361 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92584965 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1518755736911872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1510949902635008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334793454553088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334793435940864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92574988 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1518755719872512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¶¨ŸÃ(Î ûO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17516361 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92584965 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1518755736911872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1510949902635008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334793454553088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334793435940864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92574988 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1518755719872512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10565629651456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10565629651456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10684062886400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10565288312320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21519282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1529321008184832 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171804811 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚8518.535404914 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1529321366563328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79219846 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79219846 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1521499064062771 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92574988 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10549161427763 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9999 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345477603919360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1521499421652021 _step‚821 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17516358 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10684149366272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10565288312320 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345477498827264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1529321366563328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79219844 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169µ¯¢2Ï - _wandbruntime‚8519  _runtime‚8519™ü‡'Ð ÿM QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39039158 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1521646585788365 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1518902977941504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79227478 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1518902960902144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10566642285568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345512054275072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1518902977941504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39039427 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1511096413306880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10566300939264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79227478 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92584040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79227476 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1529469261841408 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1518902960902144 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1511096430346240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17518103 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10685170963456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10550172481485 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171811516 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171811516 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79227476 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10566300939264 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171821509 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92594031 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1521646943377727 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21521321 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21521055 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171821509 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10566642285568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10550513031487îåLÃöÑ „N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92594031 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1529469620227072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39039158 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1521646585788365 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1518902977941504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79227478 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1518902960902144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10566642285568 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345512054275072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1518902977941504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39039427 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1511096413306880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10566300939264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79227478 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92584040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79227476 T?DeviceStatsMonitor.on_train_batch_eÁFè%nd/allocated_bytes.all.freed‚1529469261841408 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1518902960902144 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1511096430346240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17518103 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10685170963456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10550172481485 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171811516 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171811516 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79227476 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10566300939264 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171821509 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92594031 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1521646943377727 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21521321 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21521055 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171821509 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10566642285568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10550513031487 Q&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3012412488460541 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13809683918952942 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6314393877983093 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04367409646511078 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022845778614282608 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6767147183418274 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4978611469268799 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7896454334259033 'particle/f_score‚0.6294388771057129 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22495347261428833 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3773045837879181 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7898262739181519 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23768390715122223 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20410573482513428 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7585270404815674 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5108365416526794 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6510531306266785 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7775887846946716 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7389614582061768 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6986531615257263 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.752057671546936 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10691354423761368 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.029424233362078667 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9956315159797668 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7283753156661987 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982653856277466 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7839379906654358 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2587151825428009 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972507953643799 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19581639766693115 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2605106830596924 +jet/accuracy_1_of_1‚0.33752796053886414 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6500181555747986 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7910135388374329 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4427083432674408 particle/accuracy_5_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21462509036064148 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 epoch‚24 % _timestamp‚1.7842813440938685e+09 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2605106830596924 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20903317630290985 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7898904085159302 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8546707630157471 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9275365471839905 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7802531719207764 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5645833611488342 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8561815619468689 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4963246285915375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7295532822608948 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.748479962348938 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7504750490188599 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/accuracy‚0.845059871673584 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6316863298416138 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7480042576789856 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01897389255464077 particle/accuracy_3_of_5‚1 trainer/global_step‚9999 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_3‚0.9224175214767456 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9881357550621033 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3364037871360779 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6480528116226196 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 /particle/accuracy_3_of_4‚0.9979296326637268 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12434892356395721 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.029424233362078667 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 _runtime‚8536.220319662 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_2_of_2‚1 jet/accuracy_1_of_5‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8561815619468689 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2709544003009796 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976745843887329 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964152574539185 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5329322814941406 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6990116834640503 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3011326193809509 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4894903302192688 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05403933674097061 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22141174972057343 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6695467233657837 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37544965744018555 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7876539826393127 _step‚824 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969340562820435 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894761443138123 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25084319710731506 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1204034611582756 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6085767149925232 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024835847318172455 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9944683313369751 /particle/accuracy_4_of_4‚0.9477566480636597 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21216101944446564 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9046079516410828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7123393416404724 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6400589942932129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20594124495983124 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25692781805992126 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_2_of_3‚1 6validation_average_jet_accuracy‚0.3782966732978821 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7428471446037292 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02561972290277481 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03501727059483528 +jet/accuracy_4_of_4‚0.00957049522548914 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6958246827125549 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7625094652175903 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6814990043640137 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.669414758682251 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6796875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +particle/specificity‚0.9085675477981567 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_3_of_3‚0.07277954369783401 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8668843507766724 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6802533268928528 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02874138206243515 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6387903094291687 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987532913684845 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6371615529060364 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7384135127067566 *jet/accuracy_2_of_3‚0.5745318531990051 *jet/accuracy_2_of_4‚0.7894739508628845 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4285781979560852 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7204230427742004 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6717540621757507 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7817849516868591,SÛ é{ 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6802533268928528 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02874138206243515 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6387903094291687 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987532913684845 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6371615529060364 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7384135127067566 *jet/accuracy_2_of_3‚0.5745318531990051 *jet/accuracy_2_of_4‚0.7894739508628845 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4285781979560852 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7204230427742004 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6717540621757507 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7817849516868591 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5931057333946228 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 )loss/val_total_loss‚8.001085752254346 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27219364047050476 *jet/accuracy_2_of_2‚0.1540537029504776 particle/accuracy_4_of_5‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6387903094291687 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2269873321056366 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18666666746139526 )jet/accuracy_1_of_4‚0.980448305606842 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22298313677310944 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19235806167125702 +jet/accuracy_3_of_4‚0.07902068644762039 )particle/sensitivity‚0.61282879114151 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9907498955726624 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27147674560546875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4908944070339203 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026675954461097717 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22719794511795044 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960249066352844 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7067602872848511 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951949119567871 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31767240166664124 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6253340840339661 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982653856277466 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5864822864532471 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03205074742436409 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3012412488460541 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13809683918952942 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6314393877983093 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04367409646511078 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022845778614282608 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6767147183418274 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4978611469268799 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7896454334259033 'particle/f_score‚0.6294388771057129 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22495347261428833 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3773045837879181 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7898262739181519 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23768390715122223 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20410573482513428 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7585270404815674 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5108365416526794 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6510531306266785 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.77758B<ÈÝÙ%87846946716 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7389614582061768 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6986531615257263 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.752057671546936 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10691354423761368 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.029424233362078667 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9956315159797668 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7283753156661987 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982653856277466 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7839379906654358 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2587151825428009 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972507953643799 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19581639766693115 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2605106830596924 +jet/accuracy_1_of_1‚0.33752796053886414 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6500181555747986 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7910135388374329 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4427083432674408 particle/accuracy_5_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21462509036064148 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 epoch‚24 % _timestamp‚1.7842813440938685e+09 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2605106830596924 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20903317630290985 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7898904085159302 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8546707630157471 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9275365471839905 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7802531719207764 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5645833611488342 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8561815619468689 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4963246285915375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7295532822608948 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.748479962348938 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7504750490188599 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/accuracy‚0.845059871673584 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6316863298416138 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7480042576789856 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01897389255464077 particle/accuracy_3_of_5‚1 trainer/global_step‚9999 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_3‚0.9224175214767456 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9881357550621033 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3364037871360779 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6480528116226196 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 /particle/accuracy_3_of_4‚0.9979296326637268 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12434892356395721 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.029424233362078667 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 _runtime‚8536.220319662 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_2_of_2‚1 jet/accuracy_1_of_5‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8561815619468689 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2709544003009796 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976745843887329 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964152574539185 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5329322814941406 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6990116834640503 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3011326193809509 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4894903302192688 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05403933674097061 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22141174972057343 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6695467233657837 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37544965744018555 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7876539826393127 _step‚824 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969340562820435 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894761443138123 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25084319710731506 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1204034611582756 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6085767149925232 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024835847318172455 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9944683313369751 /particle/accuracy_4_of_4‚0.9477566480636597 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21216101944446564 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9046079516410828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7123393416404724 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6400589942932129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20594124495983124 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25692781805992126 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_2_of_3‚1 6validation_average_jet_accuracy‚0.3782966732978821 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7428471446037292 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02561972290277481 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03501727059483528 +jet/accuracy_4_of_4‚0.00957049522548914 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6958246827125549 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7625094652175903 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6814990043640137 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.669414758682251 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6796875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +particle/specificity‚0.9085675477981567 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_3_of_3‚0.07277954369783401 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8668843507766724¸Oâ[Ü ‚(:’ òçҡص¢ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ózÝ ‚(:ú òçÒ‡Óò¤ disk.rbd1.in‚0.0625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 450.62890625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 407.41015625 disk.sda.in‚0$ memory_percent‚97.24957942962646 disk.sda2.in‚0 disk.dm-1.out‚ 326.01953125" disk.nvme0c0n1.in‚ 407.41015625 disk.sda.out‚ 326.01953125 network.sent‚ 1.5893403e+07* disk./.usagePercent‚36.827977150528824$ disk./.usageGB‚548.7634735107422 disk.sda3.out‚ 326.01953125 disk.rbd3.in‚0 disk.dm-0.in‚ 407.41015625 disk.drbd2165.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚36654.20458984375 disk.rbd0.in‚0 proc.cpu.threads‚17! disk.rbd1.out‚105.45263671875 disk.rbd0.out‚0 cpu‚26.050449722476852 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2812.26953125( proc.memory.percent‚17.1647310256958 network.recv‚ 2.270045e+06 disk.sda3.in‚0# disk.dm-0.out‚36654.20458984375( disk.nvme0c0n1.out‚36654.20458984375yîØ6Þ ‚(:’ œòçÒƳº¢ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚268.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.767999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÛàžöß ‚(:ë œòçÒèØ×¥ disk.sda1.in‚0( proc.memory.availableMB‚ 450.77734375 disk.nvme0n1.in‚408.125 disk.sda3.out‚ 326.22265625 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0! disk.rbd1.out‚105.57763671875 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625 network.sent‚ 1.5951126e+07$ disk./.usageGB‚548.7636337280273& disk.nvme0n1.out‚36661.07568359375 disk.sda1.out‚0 disk.nvme0c0n1.in‚408.125 cpu‚26.049863253111965 disk.drbd2165.in‚0 disk.dm-1.out‚ 326.22265625$ memory_percent‚97.24867343902588# proc.memory.rssMB‚ 2812.26953125 disk.rbd0.out‚0 disk.sda2.out‚0# disk.dm-0.out‚36661.07568359375( proc.memory.percent‚17.1647310256958 proc.cpu.threads‚17 disk.dm-0.in‚408.125 network.recv‚ 2.274946e+06* disk./.usagePercent‚36.827987902845265 disk.drbd2165.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚36661.07568359375 disk.sda.out‚ 326.22265625 disk.dm-1.in‚0qNs‰2à - _wandbruntime‚8575  _runtime‚8575ä `Êá Ä trainer/global_step‚10049 % _timestamp‚1.7842813834207854e+09 _runtime‚8575.546409521 _step‚825 ' lr-AdamW/pg1‚0.00024205356330272697 ' lr-AdamW/pg2‚0.00024205356330272697(Ê;Ïâ É _runtime‚8575.546409521 _step‚825 ' lr-AdamW/pg1‚0.00024205356330272697 ' lr-AdamW/pg2‚0.00024205356330272697 trainer/global_step‚10049 % _timestamp‚1.7842813834207854e+09¹9êÃ2ã - _wandbruntime‚8575  _runtime‚85752ÔõŸü'ä öO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10617838189790 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79722112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79722110 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347234201269248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21653454 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1538681055024128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172821872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10634058358784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1538681413402624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17606128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10753415302144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93109739 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1528047338004480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93099762 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39259585 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17606131 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚10049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93099762 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39259312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1528047355043840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336480785967104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21653184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79722110 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1538681413402624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 $ _timestamp‚1.784281383421892e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1528047338004480 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172831851 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10633717019648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10753503879168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347234308458496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79722112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1530801597082588 _step‚826 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336480804579328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1530801597082588 _step‚826 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336480804579328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1538681055024128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172821872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10634058358784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1538681413402624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17606128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10753415302144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_st_û®IUart/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93109739 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1528047338004480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93099762 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39259585 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17606131 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088ºé¼á*2æ - _wandbruntime‚8576  _runtime‚8576P÷†p'ç N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39263135 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1538829308680704 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1528194596073472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79729742 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1528194596073472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1528194579034112 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172848549 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172838556 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93108814 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347268650132992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842813840315285e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79729742 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10635070992896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17607873 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10618849243624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79729744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1538829667066368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10634729646592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1530949476397544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚8576.157412431 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21655524 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336514233337856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1530949118808182 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93118805 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21655262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1528194579034112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10618508693622N# Í 'è †N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21655262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1528194579034112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10618508693622 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39263135 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1538829308680704 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1528194596073472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79729742 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1528194596073472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1528194579034112 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172848549 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172838556 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93108814 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347268650132992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842813840315285e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79729742 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10635070992896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17607873 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10618849243624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79729744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1538829667066368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10634729646592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1530949476397544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚8576.157412431 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21655524 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336514233337856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1530949118808182 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93118805 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438» l‘2é - _wandbruntime‚8576  _runtime‚8576_õ@m ê ƒ 3loss/SRqqt1/assignment_loss‚0.36171948923861313 /loss/FRt1/detection_loss‚0.4760333621625947 .loss/FRt2/detection_loss‚1.280100372402474 _runtime‚8576.160403267 0loss/FRt1/assignment_loss‚1.6677509959730068 0loss/FRt2/assignment_loss‚0.8112333665388671 .loss/FBt1/detection_loss‚0.503724178560605 % _timestamp‚1.7842813840350497e+09 2loss/SRqqt2/assignment_loss‚0.4825511969140935 2loss/FBt1/assignment_loss‚0.008548681501028412 1loss/FBt2/assignment_loss‚0.01316682304189516 1loss/SRqqt1/detection_loss‚0.8981961510607862 1loss/SRqqt2/detection_loss‚0.6644709061898525 /loss/FBt2/detection_loss‚0.5802765501161607 %loss/total_loss‚7.747772073699977 trainer/global_step‚10049 epoch‚25 _step‚828@'ùóë ˆ 3loss/SRqqt1/assignment_loss‚0.36171948923861313 /loss/FRt1/detection_loss‚0.4760333621625947 .loss/FRt2/detection_loss‚1.280100372402474 _runtime‚8576.160403267 0loss/FRt1/assignment_loss‚1.6677509959730068 0loss/FRt2/assignment_loss‚0.8112333665388671 .loss/FBt1/detection_loss‚0.503724178560605 % _timestamp‚1.7842813840350497e+09 2loss/SRqqt2/assignment_loss‚0.4825511969140935 2loss/FBt1/assignment_loss‚0.008548681501028412 1loss/FBt2/assignment_loss‚0.01316682304189516 1loss/SRqqt1/detection_loss‚0.8981961510607862 1loss/SRqqt2/detection_loss‚0.6644709061898525 /loss/FBt2/detection_loss‚0.5802765501161607 %loss/total_loss‚7.747772073699977 trainer/global_step‚10049 epoch‚25 _step‚828¼ʬÿ8ì ‚(:’ «òçҼݤ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚269.376' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.87519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š¡wúí ‚(:ï «òçÒ‡—–¦ disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 408.953125 disk.sda.in‚0& disk.nvme0n1.out‚36669.91943359375$ memory_percent‚97.24879264831543 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 network.sent‚ 1.5955334e+07 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 408.953125! disk.rbd1.out‚105.57763671875# disk.dm-0.out‚36669.91943359375 network.recv‚ 2.277866e+06 disk.sda3.out‚ 326.4765625 disk.nvme0n1.in‚ 408.953125 cpu‚26.049338645345216$ disk./.usageGB‚548.7635688781738 disk.sda.out‚ 326.4765625 disk.sda2.out‚0 disk.dm-1.out‚ 326.4765625' proc.memory.availableMB‚ 450.7578125# proc.memory.rssMB‚ 2812.26953125) disk./.usagePercent‚36.82798355071718 disk.sda1.out‚0( disk.nvme0c0n1.out‚36669.91943359375 disk.rbd1.in‚0.0625 disk.rbd3.out‚0( proc.memory.percent‚17.1647310256958 disk.drbd2165.in‚0 disk.rbd3.in‚0f—¢­“î ‚(:ˆ ºòçÒœ–¨¢ gpu.0.gpu‚56.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06í8ôï ‚(:é ºòçÒ´­ú¤ proc.cpu.threads‚17$ disk./.usageGB‚548.7628021240234 disk.rbd0.out‚0 disk.sda2.in‚0( disk./.usagePercent‚36.8279320932028 disk.sda2.out‚0 network.sent‚ 1.6018656e+07 network.recv‚ 2.28299e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 326.65234375 disk.dm-0.in‚ 409.66796875 disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 409.66796875 disk.sda.in‚0( proc.memory.availableMB‚ 450.75390625" disk.nvme0n1.out‚ 36676.7421875$ disk.nvme0c0n1.out‚ 36676.7421875 disk.sda.out‚ 326.65234375 disk.rbd1.in‚0.0625 disk.dm-0.out‚ 36676.7421875 disk.nvme0n1.in‚ 409.66796875 disk.dm-1.out‚ 326.65234375! disk.rbd1.out‚105.64404296875# proc.memory.rssMB‚ 2812.26953125( proc.memory.percent‚17.1647310256958 disk.dm-1.in‚0 disk.rbd3.out‚0$ memory_percent‚97.24881649017334 disk.rbd3.in‚0 cpu‚26.0487687891126F’“ð ‚(:ˆ ÉòçÒòóÓ¢ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚154.691' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ð…öñ ‚(:ë ÉòçÒÏÁߤ disk.sda1.out‚0 network.recv‚ 2.286456e+06* disk./.usagePercent‚36.827935421300744$ disk./.usageGB‚548.7628517150879 disk.sda.in‚0 disk.rbd3.out‚0( proc.memory.percent‚17.1647310256958 disk.nvme0n1.in‚ 410.49609375 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 326.953125 disk.rbd1.in‚0.0625% disk.nvme0n1.out‚36685.5341796875 disk.rbd0.in‚0 disk.sda.out‚ 326.953125 disk.sda2.in‚0 disk.dm-1.out‚ 326.953125$ memory_percent‚97.24884033203125" proc.memory.availableMB‚450.75 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 410.49609375' disk.nvme0c0n1.out‚36685.5341796875 disk.dm-0.in‚ 410.49609375" disk.dm-0.out‚36685.5341796875 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2812.26953125 network.sent‚ 1.6023667e+07 disk.drbd2165.out‚0 disk.dm-1.in‚0! disk.rbd1.out‚105.64404296875 cpu‚26.0481945673114662{TO2ò - _wandbruntime‚8615  _runtime‚8615WúØûÇó Á & lr-AdamW/pg1‚0.0002401599099973967 & lr-AdamW/pg2‚0.0002401599099973967 trainer/global_step‚10099 % _timestamp‚1.7842814231538649e+09 _runtime‚ 8615.27947107 _step‚829ñDÌô Æ _runtime‚ 8615.27947107 _step‚829 & lr-AdamW/pg1‚0.0002401599099973967 & lr-AdamW/pg2‚0.0002401599099973967 trainer/global_step‚10099 % _timestamp‚1.7842814231538649e+09½ɺ>U2õ - _wandbruntime‚8615  _runtime‚8615@̺ö ÷O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1527509650075648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10684753860608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1527509633036288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93554214 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93564191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1535409406525440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39450848 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80103712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10668454676698 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338152223893504 UGDeviceStatsMonitor.on_train_batch_start/re€ºŽüquested_bytes.small_pool.peak‚ 553967983 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1535409389486080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348956934556672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21757470 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1546093802007552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338152242505728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173657924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80103710 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1535409406525440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80103712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1546094160386048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173667903 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348957039648768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10804797143040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17693381 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39451108 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842814231549559e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1546093802007552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _step‚830qܲ|(÷ üO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1546094160386048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173667903 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348957039648768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10804797143040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17693381 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39451108 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842814231549559e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1546093802007552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _step‚830 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1527509650075648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10684753860608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1527509633036288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93554214 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93564191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1535409406525440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39450848 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80103712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10668454676698 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338152223893504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1535409389486080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348956934556672 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21757470 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1546093802007552 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338152242505728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173657924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80103710 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1535409406525440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80103712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508¾KNr+2ø - _wandbruntime‚8615  _runtime‚8615¹qÛa'ù N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10685766494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348991494458880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93573257 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1535556630515712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173684601 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80111342 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39454924 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173684601 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1538325268888690 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10669125180530 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39454661 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338185671264256 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1527656143708160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93573257 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21759798 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1546242414049792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80111344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10805823194624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1535556630515712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80111344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438ƒá% ¦ú †N E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10685425148416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93563266 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1535556630515712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173684601 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80111342 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39454924 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173684601 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1538325268888690 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10669125180530 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39454661 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338185671264256 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1527656143708160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93573257 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21759798 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1546242414049792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80111344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10805823194624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1535556630515712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80111344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10685766494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348991494458880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚1215Ψ¥$f 4 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93573257 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FM ý † /loss/FRt1/assignment_loss‚1.588516756429877 1loss/SRqqt1/detection_loss‚0.9017363845949433 /loss/FBt1/detection_loss‚0.4064346485238934 /loss/FBt2/detection_loss‚1.2762881020926649 _runtime‚8615.897626036 3loss/SRqqt2/assignment_loss‚0.47493253142060876 1loss/SRqqt2/detection_loss‚0.6411686371275909 % _timestamp‚1.7842814237722888e+09 _step‚832 0loss/FRt2/assignment_loss‚0.8561589365655725 2loss/SRqqt1/assignment_loss‚0.4604754372387121 1loss/FBt2/assignment_loss‚0.09140884206114479 1loss/FBt1/assignment_loss‚0.01601171817127129 .loss/FRt1/detection_loss‚0.601212442919258 .loss/FRt2/detection_loss‚1.218291280732091 %loss/total_loss‚8.532635717877628 epoch‚25 trainer/global_step‚10099ÀYÚ:Ô“þ ‚(:ˆ ØòçÒùô’£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´×¢\ÿ ‚(:õ Øòçқ쟥 disk.dm-1.out‚ 327.1484375 network.sent‚ 1.6028706e+07! disk.rbd1.out‚105.72216796875 disk.drbd2165.out‚0# memory_percent‚97.2489595413208( disk.nvme0c0n1.out‚36690.38427734375$ disk./.usageGB‚548.7629051208496 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 410.49609375 cpu‚26.047617856184214 disk.dm-0.in‚ 410.49609375( proc.memory.availableMB‚ 450.73046875 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0# disk.dm-0.out‚36690.38427734375( proc.memory.percent‚17.1647310256958 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 327.1484375 disk.rbd1.in‚0.0625) disk./.usagePercent‚36.82793900540622 disk.sda.out‚ 327.1484375 disk.rbd3.out‚0 disk.nvme0n1.in‚ 410.49609375& disk.nvme0n1.out‚36690.38427734375 disk.sda1.in‚0 network.recv‚ 2.289548e+06 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2812.26953125€oJž€!‚(:“ çòçÒ¯ÈÏ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—jjiý!‚(:ò çòçÒ«¬î¤! disk.rbd1.out‚105.72216796875 disk.dm-1.in‚0 disk.dm-1.out‚ 327.4609375 disk.rbd1.in‚0.0625( disk.nvme0c0n1.out‚36701.95458984375 network.recv‚ 2.294735e+06 disk.nvme0n1.in‚ 411.2109375 disk.sda3.out‚ 327.4609375# proc.memory.rssMB‚ 2812.26953125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚ 327.4609375# disk.dm-0.out‚36701.95458984375) disk./.usagePercent‚36.82780664951102 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 1.6091873e+07$ disk./.usageGB‚548.7609329223633 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 411.2109375& disk.nvme0n1.out‚36701.95458984375 disk.rbd0.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 450.72265625 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 411.2109375$ memory_percent‚97.24900722503662( proc.memory.percent‚17.1647310256958 cpu‚26.04701996154552 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0²Ó4Þ“‚!‚(:ˆ öòçÒ¼Äâ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Âþƒ!‚(:ó öòçÒ—œ¥ disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0$ memory_percent‚97.24757671356201 cpu‚26.046449497322136 disk.sda.out‚ 327.63671875 disk.drbd2165.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.82781740182746$ disk./.usageGB‚548.7610931396484! disk.rbd1.out‚105.78466796875# proc.memory.rssMB‚ 2812.26953125! disk.nvme0c0n1.in‚ 412.0390625 network.sent‚ 1.6096924e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 327.63671875 disk.dm-1.in‚0 disk.dm-0.in‚ 412.0390625' disk.nvme0c0n1.out‚36709.1865234375 network.recv‚ 2.297841e+06" disk.dm-0.out‚36709.1865234375 disk.nvme0n1.in‚ 412.0390625 disk.sda1.out‚0 disk.drbd2165.in‚0( proc.memory.percent‚17.1647310256958 disk.dm-1.out‚ 327.63671875 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625% disk.nvme0n1.out‚36709.1865234375( proc.memory.availableMB‚ 450.95703125H¡Ÿì2„!- _wandbruntime‚8655  _runtime‚8655´ÆKÊ…!Ä _runtime‚8655.160582766 _step‚833 ' lr-AdamW/pg1‚0.00023826601096788675 ' lr-AdamW/pg2‚0.00023826601096788675 trainer/global_step‚10149 % _timestamp‚1.7842814630349092e+09U“/øφ!É trainer/global_step‚10149 % _timestamp‚1.7842814630349092e+09 _runtime‚8655.160582766 _step‚833 ' lr-AdamW/pg1‚0.00023826601096788675 ' lr-AdamW/pg2‚0.00023826601096788675Á<Ö¦2‡!- _wandbruntime‚8655  _runtime‚8655+$þ‰ü'ˆ!öO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1553506548990976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94008666 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17780631 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339823680432128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7842814630360305e+09 _runtime‚ 8655.16183469 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10735449362432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80485312 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1542771458007040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10735449362432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350679614929920 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1534835166629888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1542771440967680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94008666 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10719071163606 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80485310 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10856047978496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1542771440967680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1553506907369472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80485312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21861496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174493976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10735108023296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1553506907369472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10855953110016 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10735108023296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174503955 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1542771458007040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21861778 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17780628 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39642124 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94018643 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1545554254832854 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1534835183669248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1553506548990976 óDG(‰!ûO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339823680432128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7842814630360305e+09 _runtime‚ 8655.16183469 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10735449362432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80485312 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1542771458007040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10735449362432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350679614929920 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1534835166629888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1542771440967680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94008666 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10719071163606 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80485310 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10856047978496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1542771440967680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1553506907369472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80485312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21861496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174493976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10735108023296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1553506907369472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10855953110016 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10735108023296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174503955 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1542771458007040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21861778 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17780628 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39642124 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94018643 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1545554254832854 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1534835183669248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1553506548990976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1553506548990976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94008666 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17780631 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872¾"JØ2Š!- _wandbruntime‚8655  _runtime‚8655….Ó§ì ‹!ÿM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10857068901888 trainer/global_step‚10149 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1542918699036672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1542918681997312 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1542918699036672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80492944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94027709 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1553655161033216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚835 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10736461996544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174510660 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1545701418969198 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 # _timestamp‚1.78428146365661e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350714178092544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17782373 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21863586 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1534981694341120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10720082217440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10736120650240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39645959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17782376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94027709 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80492944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174520653 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80492942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10736461996544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174510660 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94017718 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10719741667438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1553655161033216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39646232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339857090578432 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1542918699036672 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1542918681997312 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1542918699036672 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80492944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94027709 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1553655161033216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚835 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10736461996544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174510660 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1545701418969198 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 # _timestamp‚1.78428146365661e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350714178092544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17782373 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21863586 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1534981694341120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10720082217440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10736120650240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39645959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17782376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94027709 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80492944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174520653 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80492942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10736461996544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174510660 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94017718 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10719741667438ÃEã Â2!- _wandbruntime‚8655  _runtime‚8655ŠoÆN Ž!† /loss/FRt2/detection_loss‚1.1055464688333565 /loss/FBt2/detection_loss‚0.7916630044381953 _step‚836 2loss/SRqqt1/assignment_loss‚0.3783335181591756 2loss/FBt1/assignment_loss‚0.007760641240980957 1loss/FBt2/assignment_loss‚0.16097666455376233 /loss/FRt1/detection_loss‚0.5114085817171218 0loss/FBt1/detection_loss‚0.31444130013197924 epoch‚25 0loss/FRt1/assignment_loss‚1.5084201435630138 0loss/FRt2/assignment_loss‚0.7569454204149125 1loss/SRqqt1/detection_loss‚0.8055942120198161 % _timestamp‚1.7842814636584003e+09 2loss/SRqqt2/assignment_loss‚0.4698631801540539 1loss/SRqqt2/detection_loss‚0.6164949446454585 &loss/total_loss‚7.4274480798718265 trainer/global_step‚10149 _runtime‚8655.783746903¢óïû!‹ 2loss/SRqqt1/assignment_loss‚0.3783335181591756 2loss/FBt1/assignment_loss‚0.007760641240980957 1loss/FBt2/assignment_loss‚0.16097666455376233 /loss/FRt1/detection_loss‚0.5114085817171218 0loss/FBt1/detection_loss‚0.31444130013197924 epoch‚25 0loss/FRt1/assignment_loss‚1.5084201435630138 0loss/FRt2/assignment_loss‚0.7569454204149125 1loss/SRqqt1/detection_loss‚0.8055942120198161 % _timestamp‚1.7842814636584003e+09 2loss/SRqqt2/assignment_loss‚0.4698631801540539 1loss/SRqqt2/detection_loss‚0.6164949446454585 &loss/total_loss‚7.4274480798718265 trainer/global_step‚10149 _runtime‚8655.783746903 /loss/FRt2/detection_loss‚1.1055464688333565 /loss/FBt2/detection_loss‚0.7916630044381953 _step‚836Ä“P«Î!‚(:’ …óçÒÐÙì¢ gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚262.071' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0râ÷º‘!‚(:ö …óçÒ©Ùí¤ disk.sda3.in‚0 disk.nvme0n1.in‚ 412.75390625 cpu‚26.045911583303045 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.sda1.in‚0 network.recv‚ 2.301345e+06 disk.dm-1.in‚0! disk.rbd1.out‚105.78466796875 disk.rbd0.in‚0( proc.memory.percent‚17.1647310256958 disk.rbd0.out‚0% disk.nvme0n1.out‚36717.7958984375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2812.26953125 network.sent‚ 1.6101941e+07 disk.dm-0.in‚ 412.75390625" disk.dm-0.out‚36717.7958984375$ memory_percent‚97.24814891815186) disk./.usagePercent‚36.82789727617815 disk.sda3.out‚ 327.86328125 disk.dm-1.out‚ 327.86328125$ disk./.usageGB‚548.7622833251953 disk.sda.out‚ 327.86328125( proc.memory.availableMB‚ 450.86328125" disk.nvme0c0n1.in‚ 412.75390625' disk.nvme0c0n1.out‚36717.7958984375 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0¡sú“’!‚(:ˆ ”óçÒ±€á¢ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚196.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.3806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PA“!‚(:õ ”óçÒ×’ç¤$ memory_percent‚97.24664688110352& proc.memory.availableMB‚ 451.109375 proc.cpu.threads‚17* disk./.usagePercent‚36.827899324238416 disk.sda2.in‚0 disk.sda3.out‚ 328.01171875$ disk./.usageGB‚548.7623138427734 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 412.80078125% disk.nvme0n1.out‚36720.7607421875 disk.sda1.out‚0# proc.memory.rssMB‚ 2812.26953125( proc.memory.percent‚17.1647310256958 network.recv‚ 2.306545e+06 disk.dm-1.out‚ 328.01171875 disk.rbd3.in‚0" disk.dm-0.out‚36720.7607421875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 cpu‚26.045346196068117 network.sent‚ 1.6165173e+07" disk.nvme0c0n1.in‚ 412.80078125 disk.sda.out‚ 328.01171875! disk.rbd1.out‚105.84716796875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚36720.7607421875 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 412.80078125 disk.drbd2165.out‚0ô?žï2”!- _wandbruntime‚8695  _runtime‚8695‡X€Ê•!Ä ' lr-AdamW/pg2‚0.00023637198785579256 trainer/global_step‚10199 % _timestamp‚1.7842815029747663e+09 _runtime‚8695.100366256 _step‚837 ' lr-AdamW/pg1‚0.00023637198785579256’bEÏ–!É _step‚837 ' lr-AdamW/pg1‚0.00023637198785579256 ' lr-AdamW/pg2‚0.00023637198785579256 trainer/global_step‚10199 % _timestamp‚1.7842815029747663e+09 _runtime‚8695.100366256Å€H¥2—!- _wandbruntime‚8695  _runtime‚8695Gàû'˜!õO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80866910 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚ 8695.10160257 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1560919295974400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94463118 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10907207498240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94473095 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 trainer/global_step‚10199 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1542160717262848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94473095 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39833654 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1552930047324112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1560919295974400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10786144864256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10786144864256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352402307244544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10769687650514 $ _timestamp‚1.784281502975812e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1550133509488640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1542160700223488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10769347100624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1550133492449280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80866910 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10907291880960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1552930404913362§ôêü™!úO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94473095 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39833654 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1552930047324112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1560919295974400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10786144864256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10786144864256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352402307244544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10769687650514 $ _timestamp‚1.784281502975812e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1550133509488640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1542160700223488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10769347100624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1550133492449280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80866910 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10907291880960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1552930404913362 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80866910 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚ 8695.10160257 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1560919295974400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94463118 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10907207498240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94473095 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 trainer/global_step‚10199 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1542160717262848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496ƼKÛ2š!- _wandbruntime‚8695  _runtime‚8695¹’ p'›!N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94482161 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80874544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94472170 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1550280750518272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1561067908016640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10787157498368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1561067549630976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1550280733478912 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10786816152064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175356705 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17869623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80874542 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10908316439040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175346712 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1561067908016640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _runtime‚8695.712485113 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341528547117056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1553077569049706 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21967822 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39837445 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17869626 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352436863556096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7842815035866523e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚839 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94482161 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39837445 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17869626 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352436863556096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7842815035866523e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚839 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94482161 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80874544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94472170 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1550280750518272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1561067908016640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10787157498368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1561067549630976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1550280733478912 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10786816152064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175356705 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17869623 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80874542 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10908316439040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175346712 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1561067908016640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _runtime‚8695.712485113 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341528547117056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1553077569049706 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21967822 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1568332042957824 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1549486250856448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10836499026944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22069324 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40024209 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354125021071360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10820304137422 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176176059 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10819963587532 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1568332042957824 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10836499026944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22069324 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40024209 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354125021071360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10820304137422 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176176059 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10819963587532 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1568332042957824 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1568332042957824 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1549486250856448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1557642784960512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343199985043456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚843 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176182764 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22071326 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22071055 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1560453719130214 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40028202 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176182764 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81256144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10820974641254 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10837511653888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81256142 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1568480296614400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10821315191256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10837853000192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 $ _timestamp‚1.784281543281014e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1557642801999872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94936613 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10959512448512 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1549632744488960 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 trainer/global_step‚10249 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1568480655000064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176192757 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40027928 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1557642784960512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343199985043456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚843 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176182764 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22071326 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22071055 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1560453719130214 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40028202 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176182764 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81256144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10820974641254 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10837511653888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81256142 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1568480296614400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10821315191256 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10837853000192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 $ _timestamp‚1.784281543281014e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1557642801999872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94936613 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10959512448512 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1549632744488960 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 trainer/global_step‚10249 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1568480655000064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176192757 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40027928 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1556811767410688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22172781 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11009830097408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1564857612451840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1564857612451840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355847723410944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1564857595412480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11009747811840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚846 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177002132 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40215162 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18042378 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1575744789941248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10870580074440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1567682347485128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95381999 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355847824308736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344837994211328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1564857595412480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10887194528768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81630112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177012111 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95372022 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1575744789941248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1575745148319744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1575745148319744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚10299 # _timestamp‚1.78428158255644e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 _runtime‚8774.682519166 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18042381 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10870920624330 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10887194528768 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344837975599104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1567682705074378 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 _runtime‚8774.682519166 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18042381 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10870920624330 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10887194528768 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344837975599104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1567682705074378 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1556811767410688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22172781 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11009830097408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1564857612451840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1564857612451840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355847723410944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1564857595412480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11009747811840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚846 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177002132 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40215162 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18042378 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1575744789941248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10870580074440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1567682347485128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95381999 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355847824308736ÎêÞ¤“¼!‚(:ˆ îóçÒ¨ƒµ¢ gpu.0.gpu‚82.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚189.042' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓS¼î½!‚(:ã îóçÒñÛº¥ disk.rbd0.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚106.14501953125 proc.cpu.threads‚17* disk./.usagePercent‚36.828019391771974 disk.nvme0n1.in‚ 416.66796875 disk.sda.in‚0# proc.memory.rssMB‚ 2812.26953125% disk.nvme0n1.out‚36771.3388671875 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 416.66796875$ memory_percent‚97.24898338317871' proc.memory.availableMB‚ 450.7265625 network.recv‚ 2.330592e+06 disk.sda1.in‚0 disk.sda.out‚329.75 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 416.66796875# disk./.usageGB‚548.764102935791 disk.rbd0.out‚0 disk.dm-1.out‚329.75 disk.sda2.in‚0 disk.sda3.out‚329.75 cpu‚26.042045284348234' disk.nvme0c0n1.out‚36771.3388671875 network.sent‚ 1.6311601e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚36771.3388671875( proc.memory.percent‚17.1647310256958ÿ›:2¾!- _wandbruntime‚8775  _runtime‚8775"µß'¿!N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18044126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81637744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10888548502016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚10299 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95391065 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10888207155712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11010856126464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10888548502016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8775.296915354 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1575893401983488 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1575893043597824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355882171914240 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1565004853481472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95381074 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177028809 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1567830226800084 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚847 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177028809 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10871591128162 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842815831706114e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81637742 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95381074 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11010767556608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81637742 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22174837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18044123 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40218719 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚344871422969856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1565004836442112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0bo 'À!†N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40218719 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚344871422969856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1565004836442112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18044126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81637744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10888548502016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚10299 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95391065 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10888207155712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11010856126464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10888548502016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8775.296915354 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1575893401983488 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1575893043597824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355882171914240 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1565004853481472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95381074 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177028809 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1567830226800084 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _step‚847 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177028809 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10871591128162 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842815831706114e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81637742 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95381074 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11010767556608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81637742 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22174837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18044123 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976Ïë.@ü2Á!- _wandbruntime‚8775  _runtime‚8775/H Â!† /loss/FRt2/detection_loss‚1.1606366292643648 % _timestamp‚1.7842815831738327e+09 2loss/SRqqt2/assignment_loss‚0.5057258208666034 2loss/FBt2/assignment_loss‚0.036485578521442366 1loss/SRqqt1/detection_loss‚0.8709838840172587 _runtime‚ 8775.29916091 2loss/SRqqt1/assignment_loss‚0.4471129589742667 1loss/SRqqt2/detection_loss‚0.5789057242236595 /loss/FBt2/detection_loss‚0.7995105641894042 _step‚848 2loss/FBt1/assignment_loss‚0.002170062398869949 0loss/FRt1/detection_loss‚0.49233956166285753 /loss/FBt1/detection_loss‚0.2604321399539913 &loss/total_loss‚7.6254489867055115 epoch‚25 trainer/global_step‚10299 0loss/FRt1/assignment_loss‚1.5879208116304306 0loss/FRt2/assignment_loss‚0.8832252510023623eñ òÃ!‹ 2loss/SRqqt2/assignment_loss‚0.5057258208666034 2loss/FBt2/assignment_loss‚0.036485578521442366 1loss/SRqqt1/detection_loss‚0.8709838840172587 _runtime‚ 8775.29916091 2loss/SRqqt1/assignment_loss‚0.4471129589742667 1loss/SRqqt2/detection_loss‚0.5789057242236595 /loss/FBt2/detection_loss‚0.7995105641894042 _step‚848 2loss/FBt1/assignment_loss‚0.002170062398869949 0loss/FRt1/detection_loss‚0.49233956166285753 /loss/FBt1/detection_loss‚0.2604321399539913 &loss/total_loss‚7.6254489867055115 epoch‚25 trainer/global_step‚10299 0loss/FRt1/assignment_loss‚1.5879208116304306 0loss/FRt2/assignment_loss‚0.8832252510023623 /loss/FRt2/detection_loss‚1.1606366292643648 % _timestamp‚1.7842815831738327e+09Ðm(pžÄ!‚(:“ ýóçÒ”Š²¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ÃÕ`òÅ!‚(:ç ýóçÒÎö¤ disk.dm-1.out‚ 329.90234375 cpu‚26.041496126027994# disk./.usageGB‚548.764045715332 disk.rbd3.in‚0 disk.sda3.out‚ 329.90234375$ memory_percent‚97.24960327148438$ disk.nvme0n1.out‚36781.173828125! disk.rbd1.out‚106.14501953125# proc.memory.availableMB‚450.625 network.recv‚ 2.335899e+06 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 417.390625( proc.memory.percent‚17.1647310256958 proc.cpu.threads‚17 disk.dm-0.in‚ 417.390625 disk.nvme0n1.in‚ 417.390625 disk.sda3.in‚0! disk.dm-0.out‚36781.173828125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2812.26953125 network.sent‚ 1.6363957e+07) disk./.usagePercent‚36.82801555165896 disk.sda.in‚0 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚36781.173828125 disk.sda.out‚ 329.90234375 disk.sda2.in‚0ÊÖÿ“Æ!‚(:ˆ ŒôçÒÞÛª¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ú]êÇ!‚(:ß ŒôçÒðˆè¥ network.sent‚ 1.6416651e+07 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 disk.sda.out‚ 330.34375 disk.sda3.out‚ 330.34375 disk.dm-1.in‚0! disk.dm-0.out‚36788.958984375 disk.sda1.in‚0 disk.rbd0.out‚0( proc.memory.percent‚17.1647310256958 network.recv‚ 2.340839e+06 disk.rbd0.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 450.859375) disk./.usagePercent‚36.82800326329732 disk.sda2.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚548.7638626098633 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 418.21875 disk.nvme0c0n1.in‚ 418.21875$ memory_percent‚97.24817276000977 cpu‚26.040940165517682 disk.sda3.in‚0 disk.dm-1.out‚ 330.34375! disk.rbd1.out‚106.20751953125$ disk.nvme0n1.out‚36788.958984375 disk.sda1.out‚0& disk.nvme0c0n1.out‚36788.958984375 disk.nvme0n1.in‚ 418.21875# proc.memory.rssMB‚ 2812.26953125 š>·2È!- _wandbruntime‚8814  _runtime‚8814¶øM5ÇÉ!Á & lr-AdamW/pg1‚0.0002306903905126254 & lr-AdamW/pg2‚0.0002306903905126254 trainer/global_step‚10349 $ _timestamp‚1.784281622538018e+09 _runtime‚8814.663631845 _step‚849@P¾ÌÊ!Æ & lr-AdamW/pg1‚0.0002306903905126254 & lr-AdamW/pg2‚0.0002306903905126254 trainer/global_step‚10349 $ _timestamp‚1.784281622538018e+09 _runtime‚8814.663631845 _step‚849Ñí§ý2Ë!- _wandbruntime‚8814  _runtime‚8814­eÉu Ì!÷O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1575058497565636 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82011710 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1583157895303168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842816225391033e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1583157536924672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1583157536924672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10938231369728 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11061164456448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18129631 _runtime‚8814.664897184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40404841 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95826474 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95826474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11061077976576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22275483 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346509432137728 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95836451 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18129628 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10921537111238 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10938231369728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10937890030592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1572219663933440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚346509413525504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1572219646894080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1564137301004288 trainer/global_step‚10349 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1572219646894080 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177838184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177848163 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357570596594176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚850 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82011712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82011710 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1564137318043648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1572219663933440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22275213 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95836451 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10937890030592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1575058855154886Zü~ê(Í!üO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10937890030592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1575058855154886 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1575058497565636 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82011710 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1583157895303168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842816225391033e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1583157536924672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1583157536924672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10938231369728 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11061164456448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18129631 _runtime‚8814.664897184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40404841 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95826474 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95826474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11061077976576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22275483 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346509432137728 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95836451 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18129628 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10921537111238 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10938231369728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10937890030592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1572219663933440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚346509413525504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1572219646894080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1564137301004288 trainer/global_step‚10349 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1572219646894080 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177838184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177848163 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357570596594176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚850 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82011712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82011710 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1564137318043648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1572219663933440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22275213 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95836451 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169Ò3l02Î!- _wandbruntime‚8815  _runtime‚8815ÊÂFQ'Ï!€N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11062111848448 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10922548165072 % _timestamp‚1.7842816231515741e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1575206376880592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8815.277505167 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95835526 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22277571 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82019344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18131373 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82019344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357605050828800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10938902657536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18131376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177864861 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177864861 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95845517 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1583306148966912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40408703 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1564283811676160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95845517 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10922207615070 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177864861 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177864861 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95845517 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1583306148966912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40408703 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1564283811676160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95845517 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10922207615070 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8815.277505167 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95835526 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UHDeviceStatsMonitor©õA½ð.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22277571 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82019344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18131373 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82019344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357605050828800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10938902657536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18131376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1590570283908096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1582434647646144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10988585532416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82393312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348180870064128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96290903 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18216881 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚10399 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚8854.394031335 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1579581698375680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359293383064064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10972153598146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10988926871552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1579581715415040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1582435005235394 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178684215 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96280926 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11112512999936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22378410 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178674236 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10988926871552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1571462851637248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1579581715415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720dŽ‘e(ß!üO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178684215 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96280926 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11112512999936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22378410 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178674236 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10988926871552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1571462851637248 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1579581715415040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1590570283908096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1582434647646144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10988585532416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82393312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348180870064128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96290903 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18216881 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚10399 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚8854.394031335 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1579581698375680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359293383064064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10972153598146 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10988926871552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1579581715415040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1582435005235394ÖÁƒ"˜2à!- _wandbruntime‚8855  _runtime‚8855ÚiÝTá!€N _runtime‚ 8855.00650657 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40598852 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStats5õKŲ%Monitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1590718895950336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359327837437952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348214280210432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1571609345269760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579728956444672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18218626 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842816628805299e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22380472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178700913 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10973164651980 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96299969 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579728939405312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96299969 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1579728939405312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11113538615296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1590718537564672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96289978 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1582582169371738 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1582582526961100 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚855 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82400942 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348214298822656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10399õ¥R² 'â!…N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359327837437952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚348214280210432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1571609345269760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579728956444672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18218626 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842816628805299e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22380472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178700913 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10973164651980 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96299969 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579728939405312 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96299969 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1579728939405312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11113538615296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1590718537564672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96289978 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1582582169371738 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1582582526961100 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚855 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82400942 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348214298822656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10399 _runtime‚ 8855.00650657 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40598852 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1590718895950336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616×µô»Ò2ã!- _wandbruntime‚8855  _runtime‚8855µáä!ˆ 0loss/FRt1/assignment_loss‚1.5309516003046335 1loss/SRqqt1/assignment_loss‚0.515349934547534 2loss/FBt1/assignment_loss‚0.017022931988471622 1loss/SRqqt2/detection_loss‚0.6327012984292003 %loss/total_loss‚7.645465578517172 _runtime‚8855.008225013 0loss/FBt1/detection_loss‚0.32208907477341703 0loss/FRt2/assignment_loss‚0.7614305901897046 3loss/SRqqt2/assignment_loss‚0.42762332941239745 /loss/FRt1/detection_loss‚0.5317569301853026 /loss/FRt2/detection_loss‚1.2495305138629276 1loss/SRqqt1/detection_loss‚0.9495773773849392 /loss/FBt2/detection_loss‚0.7073108264765537 trainer/global_step‚10399 _step‚856 4loss/FBt2/assignment_loss‚0.00012117096209084867 epoch‚25 % _timestamp‚1.7842816628828743e+097%ìzå! 0loss/FRt1/assignment_loss‚1.5309516003046335 1loss/SRqqt1/assignment_loss‚0.515349934547534 2loss/FBt1/assignment_loss‚0.017022931988471622 1loss/SRqqt2/detection_loss‚0.6327012984292003 %loss/total_loss‚7.645465578517172 _runtime‚8855.008225013 0loss/FBt1/detection_loss‚0.32208907477341703 0loss/FRt2/assignment_loss‚0.7614305901897046 3loss/SRqqt2/assignment_loss‚0.42762332941239745 /loss/FRt1/detection_loss‚0.5317569301853026 /loss/FRt2/detection_loss‚1.2495305138629276 1loss/SRqqt1/detection_loss‚0.9495773773849392 /loss/FBt2/detection_loss‚0.7073108264765537 trainer/global_step‚10399 _step‚856 4loss/FBt2/assignment_loss‚0.00012117096209084867 epoch‚25 % _timestamp‚1.7842816628828743e+09ØÉÿIä’æ!‚(:‡ ÈôçÒº€ù¢ gpu.0.gpu‚40.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0- äRïç!‚(:ä ÈôçÒ£ºþ¥ disk.sda.in‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚36815.26953125 network.sent‚ 1.65078e+07 disk.dm-1.in‚0 disk.rbd3.in‚0# disk.nvme0n1.out‚36815.26953125 disk.drbd2165.out‚0 disk.rbd0.out‚0$ disk./.usageGB‚548.7640419006348! disk.rbd1.out‚106.39501953125$ memory_percent‚97.26946353912354 disk.sda2.in‚0 cpu‚26.043020139719776 disk.sda3.out‚ 331.34375 disk.dm-1.out‚ 331.34375 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 420.4765625# proc.memory.rssMB‚ 2815.23828125 proc.cpu.threads‚17 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.dm-0.in‚ 420.4765625 disk.dm-0.out‚36815.26953125 disk.sda1.in‚0( proc.memory.availableMB‚ 447.37109375) proc.memory.percent‚17.18285083770752 network.recv‚ 2.355676e+06* disk./.usagePercent‚36.828015295651426 disk.rbd0.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 420.4765625 disk.drbd2165.in‚0 disk.sda.out‚ 331.34375¾2è!- _wandbruntime‚8872  _runtime‚8872âsÈþ&é!Ö{ ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.649193525314331 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9375371336936951 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8944750428199768 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5399420857429504 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08504930883646011 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6595987677574158 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2103554904460907 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025111258029937744 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998410165309906 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6719563603401184 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21925704181194305 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6931952834129333 7"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.79545658826828 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857657551765442 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6927480101585388 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6668601036071777 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9281675219535828 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22404396533966064 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969547986984253 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2872065007686615 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.06701985746622086 % _timestamp‚1.7842816799994147e+09 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2885405719280243 +jet/accuracy_1_of_1‚0.35074326395988464 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9916605353355408 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025828350335359573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9867425560951233 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.969339484245621 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7842025756835938 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7766817808151245 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2774248421192169 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7779856324195862 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6854232549667358 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982649683952332 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6168110370635986 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5404165983200073 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909955263137817 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6608679890632629 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23873135447502136 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7082168459892273 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6955350637435913 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 (particle/accuracy‚0.8453929424285889 +particle/specificity‚0.9041021466255188 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9833211302757263 _runtime‚8872.125733803 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_4‚0.9863086938858032 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6608679890632629 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.607010543346405 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3919515609741211 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2463902235031128 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07281456887722015 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5834193229675293 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_2‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5246009230613708 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0518290214240551 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2750653922557831 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2722722291946411 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6374045014381409 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995595395565033 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7551843523979187 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6478501558303833 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8401374816894531 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7331892848014832 trainer/global_step‚10399 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23528601229190826 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7237488627433777 particle/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2886255383491516 &jet/accuracy_1_of_2‚0.809326171875 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_2_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23430296778678894 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2774248421192169 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9604476690292358 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7662887573242188 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963992834091187 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7523115873336792 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04465179890394211 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7358314394950867 +particle/sensitivity‚0.6306967735290527 6validation_average_jet_accuracy‚0.3906613290309906 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6572407484054565 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7920840978622437 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8626329302787781 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.030906150117516518 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947257041931152 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7028155326843262 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14271655678749084 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.46955370903015137 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7960201501846313 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3907487392425537 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04170070216059685 particle/accuracy_1_of_1‚1 'particle/f_score‚0.6366105079650879 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8667823076248169 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4444444477558136 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 >&Purity/1FRt0SRq å÷Þqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.030906150117516518 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3823161721229553 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.712134599685669 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23972299695014954 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46808668971061707 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 particle/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5081889629364014 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3506607115268707 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951376914978027 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922646880149841 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6674333810806274 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15921474993228912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7228313684463501 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8626329302787781 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3181169927120209 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7706676721572876 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5984408259391785 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33285197615623474 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23556944727897644 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7898904085159302 +jet/accuracy_2_of_2‚0.15759678184986115 _step‚857 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028044559061527252 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7977419495582581 )jet/accuracy_2_of_4‚0.784866988658905 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7163602113723755 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5056375861167908 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.277153879404068 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7404677867889404 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2435489147901535 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17352330684661865 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 epoch‚25 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08615081012248993 particle/accuracy_2_of_4‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7646172046661377 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26866117119789124 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9254559278488159 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2473241686820984 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5965425968170166 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7422366142272949 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5804430246353149 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15703782439231873 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6558966636657715 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916567325592 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2887246906757355 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25839537382125854 particle/accuracy_3_of_4‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028484078124165535 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7174649238586426 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.038237642496824265 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_3_of_3‚1¹…á=ê!Û{ 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23430296778678894 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2774248421192169 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9604476690292358 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7662887573242188 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963992834091187 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7523115873336792 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04465179890394211 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7358314394950867 +particle/sensitivity‚0.6306967735290527 6validation_average_jet_accuracy‚0.3906613290309906 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6572407484054565 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7920840978622437 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8626329302787781 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.030906150117516518 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947257041931152 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7028155326843262 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14271655678749084 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.46955370903015137 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7960201501846313 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3907487392425537 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04170070216059685 particle/accuracy_1_of_1‚1 'particle/f_score‚0.6366105079650879 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8667823076248169 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4444444477558136 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.030906150117516518 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3823161721229553 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.712134599685669 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23972299695014954 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46808668971061707 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 particle/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5081889629364014 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3506607115268707 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951376914978027 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9922646880149841 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6674333810806274 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15921474993228912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7228313684463501 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8626329302787781 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3181169927120209 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7706676721572876 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5984408259391785 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33285197615623474 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23556944727897644 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7898904085159302 +jet/accuracy_2_of_2‚0.15759678184986115 _step‚857 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028044559061527252 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7977419495582581 )jet/accuracy_2_of_4‚0.784866988658905 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7163602113723755 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5056375861167908 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.277153879404068 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7404677867889404 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2435489147901535 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17352330684661865 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 epoch‚25 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.08615081012248993 particle/accuracy_2_of_4‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7646172046661377 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26866117119789124 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9254559278488159 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2473241686820984 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5965425968170166 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7422366142272949 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5804430246353149 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15703782439231873 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6558966636657715 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916567325592 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2887246906757355 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25839537382125854 particle/accuracy_3_of_4‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028484078124165535 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7174649238586426 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.038237642496824265 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_3_of_3‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.649193525314331 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9375371336936951 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8944750428199768 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5399420857429504 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9956042766571045 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08504930883646011 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6595987677574158 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2103554904460907 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025111258029937744 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998410165309906 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6719563603401184 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21925704181194305 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6931952834129333 7"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.79545658826828 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857657551765442 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6927480101585388 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6668601036071777 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9281675219535828 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22404396533966064 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969547986984253 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2872065007686615 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.06701985746622086 % _timestamp‚1.7842816799994147e+09 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2885405719280243 +jet/accuracy_1_of_1‚0.35074326395988464 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9916605353355408 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025828350335359573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9867425560951233 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚7.969339484245621 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7842025756835938 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7766817808151245 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2774248421192169 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7779856324195862 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6854232549667358 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982649683952332 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6168110370635986 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5404165983200073 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909955263137817 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6608679890632629 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23873135447502136 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7082168459892273 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6955350637435913 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 (particle/accuracy‚0.8453929424285889 +particle/specificity‚0.9041021466255188 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9833211302757263 _runtime‚8872.125733803 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_4‚0.9863086938858032 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6608679890632629 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.607010543346405 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3919515609741211 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2463902235031128 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07281456887722015 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5834193229675293 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_2‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5246009230613708 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0518290214240551 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900226593017578 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2750653922557831 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2722722291946411 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6374045014381409 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995595395565033 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7551843523979187 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6478501558303833 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8401374816894531 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7331892848014832 trainer/global_step‚10399 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23528601229190826 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7237488627433777 particle/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2886255383491516 &jet/accuracy_1_of_2‚0.809326171875 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_2_of_3‚1Ù¯7¯žë!‚(:“ ×ôçÒÊᣣ gpu.0.gpu‚62.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r—þì!‚(:õ ×ôçÒ¬Øô¨ disk.sda1.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚36.828042944465125 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 442.45703125# proc.memory.rssMB‚ 2820.51953125 cpu‚26.045758945920124 disk.nvme0n1.in‚ 421.3046875 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 421.3046875 disk.sda3.in‚0 disk.dm-1.in‚0 network.sent‚ 1.6557109e+07! disk.rbd1.out‚106.39501953125) proc.memory.percent‚17.21508502960205 proc.cpu.threads‚17 disk.rbd0.in‚0' disk.nvme0c0n1.out‚36823.5927734375$ disk./.usageGB‚548.7644538879395 disk.sda1.out‚0% disk.nvme0n1.out‚36823.5927734375$ memory_percent‚97.29945659637451 disk.dm-1.out‚ 331.67578125 disk.dm-0.in‚ 421.3046875" disk.dm-0.out‚36823.5927734375 network.recv‚ 2.360508e+06 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 331.67578125 disk.sda.out‚ 331.67578125 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0¨jXí!‚(:’ æôçÒÓ订 gpu.0.gpu‚51.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ó•ýî!‚(:ö æôçÒ•¡Ž¥# disk.dm-0.out‚36826.65869140625 proc.cpu.threads‚17 network.sent‚ 1.6614628e+07 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 421.3046875) disk./.usagePercent‚36.82805011267609& disk.nvme0n1.out‚36826.65869140625 disk.sda.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2820.51953125 network.recv‚ 2.365547e+06 disk.rbd0.out‚0( disk.nvme0c0n1.out‚36826.65869140625 disk.rbd1.in‚0.0625 disk.rbd3.out‚0) proc.memory.percent‚17.21508502960205 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚97.30069637298584 cpu‚26.04523009653544 disk.dm-0.in‚ 421.3046875$ disk./.usageGB‚548.7645606994629 disk.nvme0n1.in‚ 421.3046875! disk.rbd1.out‚106.52001953125 disk.rbd3.in‚0( proc.memory.availableMB‚ 442.25390625 disk.drbd2165.in‚0 disk.sda.out‚ 331.92578125 disk.sda3.out‚ 331.92578125 disk.dm-1.out‚ 331.92578125KwŽ‘ï!‚(:† õôçÒÓÄú¢ gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã Òð!‚(:ø õôçÒ€¶£¥( disk.nvme0c0n1.out‚36848.04150390625 disk.sda3.in‚0 disk.sda3.out‚ 333.7109375! disk.rbd1.out‚106.52001953125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 422.01953125 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚97.29955196380615# proc.memory.rssMB‚ 2820.51953125) proc.memory.percent‚17.21508502960205 disk.sda.in‚0 disk.dm-0.in‚ 422.01953125& disk.nvme0n1.out‚36848.04150390625 disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 422.01953125 disk.rbd3.in‚0# disk.dm-0.out‚36848.04150390625 cpu‚26.044622831906846 disk.sda2.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.828365257950736 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 network.sent‚ 1.6619823e+07 network.recv‚ 2.369054e+06 disk.sda.out‚ 333.7109375 disk.dm-1.out‚ 333.7109375( proc.memory.availableMB‚ 442.44140625$ disk./.usageGB‚548.7692565917969§:¦2ñ!- _wandbruntime‚8911  _runtime‚8911ÝbXÈò! % _timestamp‚1.7842817195376167e+09 _runtime‚8911.663330262 _step‚858 & lr-AdamW/pg1‚0.0002269042686327356 & lr-AdamW/pg2‚0.0002269042686327356 trainer/global_step‚10449h.*Íó!Ç _runtime‚8911.663330262 _step‚858 & lr-AdamW/pg1‚0.0002269042686327356 & lr-AdamW/pg2‚0.0002269042686327356 trainer/global_step‚10449 % _timestamp‚1.7842817195376167e+09Ú; ‘y2ô!- _wandbruntime‚8911  _runtime‚8911뎛õ!÷O ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1588873333547008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349868220090368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11181828404224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11057355578880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11040489810283 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚859 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1599930330747392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96815677 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82895576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1588873316507648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22516714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96815677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1580697031405568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NEDeviceStatsMonitor.on_train_batch_start'zËKà/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11181741924352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179701276 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96805700 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40823365 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1591737180665961 _runtime‚8911.664655905 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40823079 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96805700 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11057014239744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18306651 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361049943402496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1591737538255211 trainer/global_step‚10449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18306648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1599930689125888ÜTf(ö!üO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96805700 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11057014239744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18306651 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361049943402496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1591737538255211 trainer/global_step‚10449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18306648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1599930689125888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1588873333547008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349868220090368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11181828404224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11057355578880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11040489810283 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚859 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1599930330747392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96815677 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82895576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1588873316507648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22516714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96815677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1580697031405568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11181741924352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179701276 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96805700 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40823365 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1591737180665961 _runtime‚8911.664655905 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40823079 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Û?b¨2÷!- _wandbruntime‚8912  _runtime‚8912p³»w'ø!N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1589020557537280 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11058368212992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179727953 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179717960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22518787 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚8912.276392225 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1589020574576640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82903210 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361084504101376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361084394822144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11041500864117 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179717960 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1589020557537280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82903208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96824743 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349901630236672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842817201505167e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179727953 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96814752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1600078942789632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11182764585472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1591885059980917 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚860 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40827183 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96824743 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11058368212992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1600078942789632 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1589020574576640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1600078584403968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1591884702391555 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40826921 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11182855252480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚10449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82903210 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96814752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22518528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11041160314115 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1580843559116800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1580843542077440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11058026866688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18308396 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349901648848896æÙ/+ ù!†N OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚8912.276392225 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1589020574576640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82903210 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361084504101376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361084394822144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11041500864117 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179717960 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1589020557537280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82903208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96824743 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349901630236672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842817201505167e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179727953 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96814752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1600078942789632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11182764585472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1591885059980917 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚860 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40827183 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96824743 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11058368212992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1600078942789632 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1589020574576640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1600078584403968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1591884702391555 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40826921 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11182855252480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚10449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82903210 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96814752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22518528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11041160314115 _JDeviceStatsMonitor.on_train_batch_end/requested_b¥1»Jytes.large_pool.allocated‚1580843559116800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1580843542077440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11058026866688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18308396 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349901648848896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1589020557537280 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11058368212992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179727953 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179717960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22518787ܽ–ä{2ú!- _wandbruntime‚8912  _runtime‚8912¡œb3 û!† %loss/total_loss‚7.721309122186036 epoch‚26 2loss/SRqqt1/assignment_loss‚0.4106826656781744 .loss/FRt1/detection_loss‚0.496779897725943 % _timestamp‚1.7842817201537066e+09 _runtime‚8912.279043029 _step‚861 0loss/FRt1/assignment_loss‚1.6365404420943193 3loss/SRqqt2/assignment_loss‚0.43605733096416266 /loss/FRt2/detection_loss‚1.2386749864433537 0loss/SRqqt1/detection_loss‚0.822995132247845 trainer/global_step‚10449 2loss/FBt1/assignment_loss‚0.014310719907368253 3loss/FBt2/assignment_loss‚4.669442810234514e-05 1loss/SRqqt2/detection_loss‚0.6779009225380018 /loss/FBt2/detection_loss‚0.7163116245064884 0loss/FRt2/assignment_loss‚0.8702353884682538 0loss/FBt1/detection_loss‚0.40077331718402326~Hiü!‹ 2loss/SRqqt1/assignment_loss‚0.4106826656781744 .loss/FRt1/detection_loss‚0.496779897725943 % _timestamp‚1.7842817201537066e+09 _runtime‚8912.279043029 _step‚861 0loss/FRt1/assignment_loss‚1.6365404420943193 3loss/SRqqt2/assignment_loss‚0.43605733096416266 /loss/FRt2/detection_loss‚1.2386749864433537 0loss/SRqqt1/detection_loss‚0.822995132247845 trainer/global_step‚10449 2loss/FBt1/assignment_loss‚0.014310719907368253 3loss/FBt2/assignment_loss‚4.669442810234514e-05 1loss/SRqqt2/detection_loss‚0.6779009225380018 /loss/FBt2/detection_loss‚0.7163116245064884 0loss/FRt2/assignment_loss‚0.8702353884682538 0loss/FBt1/detection_loss‚0.40077331718402326 %loss/total_loss‚7.721309122186036 epoch‚26ÝK¯Ka“ý!‚(:ˆ „õçÒ”»Ú¥ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Ù¸äúþ!‚(:ï „õçÒø©è§ disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 422.84375 disk.sda1.out‚0( proc.memory.availableMB‚ 442.19921875 disk.rbd3.in‚0 disk.dm-0.in‚ 422.84375 disk.drbd2165.in‚0 network.sent‚ 1.6624881e+07$ disk./.usageGB‚548.7651481628418 disk.sda3.out‚ 335.6640625 disk.dm-1.out‚ 335.6640625( disk.nvme0c0n1.out‚36860.71826171875 disk.sda.in‚0) proc.memory.percent‚17.21508502960205& disk.nvme0n1.out‚36860.71826171875$ memory_percent‚97.30103015899658 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚ 2.372304e+06* disk./.usagePercent‚36.828089537836355 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 422.84375 disk.dm-1.in‚0 disk.sda.out‚ 335.6640625 disk.rbd3.out‚0# proc.memory.rssMB‚ 2820.51953125 disk.rbd0.in‚0 disk.sda2.in‚0! disk.rbd1.out‚106.58251953125# disk.dm-0.out‚36860.71826171875 disk.sda1.in‚0 cpu‚26.044059048600854í7“ÿ!‚(:ˆ “õçÒø訢 gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç¬\ù€"‚(:î “õçÒŸÞ̤ disk.drbd2165.in‚0 disk.rbd3.out‚0$ memory_percent‚97.30257987976074 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚36873.00048828125 disk.sda2.in‚0 disk.sda1.in‚0 network.recv‚ 2.377687e+06 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 423.55859375 disk.sda.in‚0 disk.sda3.out‚336.5625! disk.rbd1.out‚106.58251953125) proc.memory.percent‚17.21508502960205 cpu‚26.043531950356165* disk./.usagePercent‚36.828089793843894 disk.sda.out‚336.5625 disk.sda3.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚36873.00048828125 disk.rbd3.in‚0# disk.dm-0.out‚36873.00048828125 disk.nvme0n1.in‚ 423.55859375' proc.memory.availableMB‚ 441.9453125 proc.cpu.threads‚17$ disk./.usageGB‚548.7651519775391 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.sda1.out‚0# proc.memory.rssMB‚ 2820.51953125 network.sent‚ 1.6687953e+07 disk.dm-1.out‚336.5625 disk.dm-0.in‚ 423.55859375} ë2"- _wandbruntime‚8951  _runtime‚8951GìbZÊ‚"Ä ' lr-AdamW/pg2‚0.00022501205539802256 trainer/global_step‚10499 % _timestamp‚1.7842817592926989e+09 _runtime‚8951.418434658 _step‚862 ' lr-AdamW/pg1‚0.00022501205539802256f¢Ã]σ"É ' lr-AdamW/pg1‚0.00022501205539802256 ' lr-AdamW/pg2‚0.00022501205539802256 trainer/global_step‚10499 % _timestamp‚1.7842817592926989e+09 _runtime‚8951.418434658 _step‚862Þä 42„"- _wandbruntime‚8951  _runtime‚8951=½q§ý'…"÷O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22620427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1607343436109312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11091106297191 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1607343077730816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11233127107072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97260152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83277178 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41014600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97270129 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1599113688335719 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚863 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362772884186624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11090765747301 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1599113330746469 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚8951.419684849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1596235367989248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1596235367989248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11233226169856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1607343077730816 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1596235385028608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97260152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83277178 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41014600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97270129 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1599113688335719 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚863 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362772884186624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11090765747301 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1599113330746469 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚8951.419684849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1596235367989248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1596235367989248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11233226169856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1607343077730816 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1596235385028608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83284808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97279195 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11092117351025 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83284810 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1596382609018880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11108722368512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18395646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18395643 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22622700 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41018067 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351573086775296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeväådÏWiceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83284810 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41018346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22622424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180554012 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97269204 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362807325299712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180564005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1607491689773056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362807213923328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11108722368512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1588169075671040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11091776801023 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351573068163072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1599261210061425 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83284808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180564005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11234145760256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842817600922065e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180554012 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0p¸ž '‰"†N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83284808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180564005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11234145760256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842817600922065e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180554012 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11109063714816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83284808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97279195 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11092117351025 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83284810 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1596382609018880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11108722368512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18395646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18395643 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22622700 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41018067 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351573086775296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83284810 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41018346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22622424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180554012 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97269204 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362807325299712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180564005 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1607491689773056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362807213923328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11108722368512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1588169075671040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11091776801023 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351573068163072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1599261210061425 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269à—{~62Š"- _wandbruntime‚8952  _runtime‚8952;ÉÐI ‹"„ 3loss/SRqqt1/assignment_loss‚0.45835661911120484 /loss/FRt2/detection_loss‚1.2648055981855097 epoch‚26 0loss/FRt2/assignment_loss‚0.8459202061284263 2loss/SRqqt2/assignment_loss‚0.4857369708517869 3loss/FBt2/assignment_loss‚0.0012142573618649788 -loss/FRt1/detection_loss‚0.54226606154581 0loss/SRqqt2/detection_loss‚0.620449831535557 /loss/FBt1/detection_loss‚0.3047493122362842 _step‚865 2loss/FBt1/assignment_loss‚0.023657043926618458 _runtime‚8952.220596838 0loss/FRt1/assignment_loss‚1.5895369404351094 1loss/SRqqt1/detection_loss‚0.9197575031999052 0loss/FBt2/detection_loss‚0.46200538652192336 %loss/total_loss‚7.518455731040001 trainer/global_step‚10499 $ _timestamp‚1.784281760095267e+09 hêŒ"‰ 0loss/FRt2/assignment_loss‚0.8459202061284263 2loss/SRqqt2/assignment_loss‚0.4857369708517869 3loss/FBt2/assignment_loss‚0.0012142573618649788 -loss/FRt1/detection_loss‚0.54226606154581 0loss/SRqqt2/detection_loss‚0.620449831535557 /loss/FBt1/detection_loss‚0.3047493122362842 _step‚865 2loss/FBt1/assignment_loss‚0.023657043926618458 _runtime‚8952.220596838 0loss/FRt1/assignment_loss‚1.5895369404351094 1loss/SRqqt1/detection_loss‚0.9197575031999052 0loss/FBt2/detection_loss‚0.46200538652192336 %loss/total_loss‚7.518455731040001 trainer/global_step‚10499 $ _timestamp‚1.784281760095267e+09 3loss/SRqqt1/assignment_loss‚0.45835661911120484 /loss/FRt2/detection_loss‚1.2648055981855097 epoch‚26áÄj1’‘"‚(:† ¢õçÒº‘«¥ gpu.0.gpu‚70.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.02' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˳µ%îŽ"‚(:ã ¢õçÒ—òܦ) proc.memory.percent‚17.21508502960205 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 424.38671875( proc.memory.availableMB‚ 442.46484375 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2820.51953125 cpu‚26.04302997181647 proc.cpu.threads‚17 network.recv‚ 2.381092e+06 disk.rbd0.out‚0$ disk./.usageGB‚548.7647857666016 disk.dm-1.out‚ 337.078125 disk.rbd1.in‚0.0625( disk./.usagePercent‚36.8280652171206 disk.sda1.in‚0 network.sent‚ 1.6693194e+07! disk.nvme0n1.out‚ 36880.328125# disk.nvme0c0n1.out‚ 36880.328125 disk.sda3.out‚ 337.078125 disk.dm-0.out‚ 36880.328125$ memory_percent‚97.29940891265869 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 424.38671875 disk.sda.in‚0! disk.rbd1.out‚106.64501953125 disk.nvme0n1.in‚ 424.38671875 disk.drbd2165.in‚0 disk.sda.out‚ 337.078125 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0¬‚™Š"‚(:’ ±õçÒ´ ¼¢ gpu.0.gpu‚49.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÅÆÿ"‚(:ô ±õçÒ¹µØ¤ disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 337.5234375" disk.dm-0.out‚36885.9638671875$ memory_percent‚97.30229377746582$ disk./.usageGB‚548.7661972045898 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2820.51953125 disk.drbd2165.out‚0 disk.dm-0.in‚ 424.38671875 network.recv‚ 2.386161e+06 disk.nvme0n1.in‚ 424.38671875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 337.5234375! disk.rbd1.out‚106.64501953125 cpu‚26.042499215199093* disk./.usagePercent‚36.828159939908275 disk.drbd2165.in‚0 disk.sda.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚36885.9638671875' proc.memory.availableMB‚ 441.9921875 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 424.38671875' disk.nvme0c0n1.out‚36885.9638671875 disk.sda.out‚ 337.5234375 disk.sda2.in‚0) proc.memory.percent‚17.21508502960205 disk.sda3.in‚0 network.sent‚ 1.6756253e+07Éþ;“‘"‚(:ˆ ÀõçÒîÝ’¢ gpu.0.gpu‚86.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôø’ü’"‚(:ñ ÀõçÒÒ‹¤¥ disk.dm-1.out‚ 338.109375 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 425.1015625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚36892.23974609375# proc.memory.rssMB‚ 2820.51953125) proc.memory.percent‚17.21508502960205 cpu‚26.041899117993868 proc.cpu.threads‚17& disk.nvme0n1.out‚36892.23974609375$ memory_percent‚97.30274677276611$ disk./.usageGB‚548.7646827697754 disk.drbd2165.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚36892.23974609375 disk.sda.out‚ 338.109375 disk.nvme0n1.in‚ 425.1015625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 338.109375 disk.dm-1.in‚0 disk.rbd3.out‚0 network.sent‚ 1.6761389e+07 disk.sda.in‚0 disk.dm-0.in‚ 425.1015625 network.recv‚ 2.389687e+06) disk./.usagePercent‚36.82805830491718! disk.rbd1.out‚106.70751953125 disk.rbd3.in‚0( proc.memory.availableMB‚ 441.91796875V»`-2“"- _wandbruntime‚8991  _runtime‚8991|'¤Ê”"Ä _runtime‚8991.329028865 _step‚866 ' lr-AdamW/pg1‚0.00022312056935752474 ' lr-AdamW/pg2‚0.00022312056935752474 trainer/global_step‚10549 % _timestamp‚1.7842817992034264e+09Hž VÏ•"É _runtime‚8991.329028865 _step‚866 ' lr-AdamW/pg1‚0.00022312056935752474 ' lr-AdamW/pg2‚0.00022312056935752474 trainer/global_step‚10549 % _timestamp‚1.7842817992034264e+09âÊ"Ëò2–"- _wandbruntime‚8991  _runtime‚8991´aÄÔü'—"öO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11284222618624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1614756183092736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11158405243392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97724581 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1606489480826977 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41204490 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1614755824714240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18481148 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353211095943168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1595348098592768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97714604 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22723339 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚8991.330306146 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1614756183092736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83658778 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11158405243392 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11141382234209 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1614755824714240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚10549 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11158746582528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11158746582528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181373380 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1603597419470848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1595348115632128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97714604 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1603597436510208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181383359 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364495207178240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784281799204502e+09 _step‚867 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83658778 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18481151 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83658776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97724581 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1603597419470848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1614755824714240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚10549 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11158746582528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11158746582528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181373380 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1603597419470848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1595348115632128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97714604 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1603597436510208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181383359 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364495207178240 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784281799204502e+09 _step‚867 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83658778 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18481151 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83658776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97724581 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1603597419470848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97714604 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22723339 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚8991.330306146 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1614756183092736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83658778 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11158405243392 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11141382234209 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ã/ŽFI2™"- _wandbruntime‚8991  _runtime‚8991mÈ„'š"N OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83666408 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97733647 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1595494626304000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1603744660500480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181400057 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11159417870336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364529648857088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97733647 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11159417870336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11142393287931 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364529764427776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1595494609264640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚868 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22725366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353244524701696 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1606637360141933 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18482896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353244506089472 trainer/global_step‚10549 _runtime‚8991.941031144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1603744677539840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181390064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41207995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1603744677539840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97723656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22725102 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842817998151917e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181400057 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83666410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11285142767616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11159759216640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181390064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993­™Ì4 '›"†N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181390064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83666408 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97733647 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1595494626304000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1603744660500480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181400057 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11159417870336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364529648857088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97733647 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11159417870336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11142393287931 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364529764427776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1595494609264640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚868 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22725366 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353244524701696 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1606637360141933 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18482896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353244506089472 trainer/global_step‚10549 _runtime‚8991.941031144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1603744677539840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181390064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41207995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1603744677539840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97723656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22725102 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842817998151917e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181400057 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83666410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11285142767616 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11159759216640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ä³í–2œ"- _wandbruntime‚8991  _runtime‚8991A¥É" 3loss/SRqqt1/assignment_loss‚0.43561860379066214 2loss/SRqqt2/assignment_loss‚0.4940468079449267 /loss/FRt1/detection_loss‚0.5409232982982518 .loss/FRt2/detection_loss‚1.265958077673487 1loss/SRqqt2/detection_loss‚0.6256632490169526 /loss/SRqqt1/detection_loss‚0.83774121185857 0loss/FBt1/detection_loss‚0.42635168580688454 /loss/FBt2/detection_loss‚0.5423252721268096 %loss/total_loss‚7.974549936588147 trainer/global_step‚10549 % _timestamp‚1.7842817998187916e+09 _runtime‚8991.944128692 epoch‚26 /loss/FRt2/assignment_loss‚1.022026415732916 1loss/FBt1/assignment_loss‚0.04359638431706523 _step‚869 1loss/FBt2/assignment_loss‚0.10839323255693838 0loss/FRt1/assignment_loss‚1.6319056974646833R†‡? ž"† 0loss/FRt1/assignment_loss‚1.6319056974646833 3loss/SRqqt1/assignment_loss‚0.43561860379066214 2loss/SRqqt2/assignment_loss‚0.4940468079449267 /loss/FRt1/detection_loss‚0.5409232982982518 .loss/FRt2/detection_loss‚1.265958077673487 1loss/SRqqt2/detection_loss‚0.6256632490169526 /loss/SRqqt1/detection_loss‚0.83774121185857 0loss/FBt1/detection_loss‚0.42635168580688454 /loss/FBt2/detection_loss‚0.5423252721268096 %loss/total_loss‚7.974549936588147 trainer/global_step‚10549 % _timestamp‚1.7842817998187916e+09 _runtime‚8991.944128692 epoch‚26 /loss/FRt2/assignment_loss‚1.022026415732916 1loss/FBt1/assignment_loss‚0.04359638431706523 _step‚869 1loss/FBt2/assignment_loss‚0.10839323255693838å>2"‚(:“ ÏõçÒɬ£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kä×(ö "‚(:ë ÏõçÒ’”ܦ% disk.nvme0c0n1.out‚36900.81640625 disk.sda.out‚ 338.50390625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2820.51953125 disk.sda1.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0.0625! disk.nvme0c0n1.in‚ 425.9296875# disk.nvme0n1.out‚36900.81640625 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.out‚0 network.sent‚ 1.6766296e+07 disk.sda2.in‚0 disk.dm-0.in‚ 425.9296875) disk./.usagePercent‚36.82799532706375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 338.50390625 disk.dm-1.out‚ 338.50390625) proc.memory.percent‚17.21508502960205 disk.nvme0n1.in‚ 425.9296875 disk.sda3.in‚0! disk.rbd1.out‚106.70751953125 disk.dm-0.out‚36900.81640625# disk./.usageGB‚548.763744354248 disk.sda1.in‚0# memory_percent‚97.3029613494873' proc.memory.availableMB‚ 441.8828125 network.recv‚ 2.393163e+06 disk.rbd3.out‚0 cpu‚26.041351063347065ÍÉC“¡"‚(:ˆ ÞõçÒü”¢ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚248.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©9¢"‚(:ú ÞõçÒ†úš¥! disk.rbd1.out‚106.83251953125 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 426.64453125( disk.nvme0c0n1.out‚36907.82275390625 disk.nvme0n1.in‚ 426.64453125 disk.sda2.in‚0 disk.dm-1.out‚ 338.81640625 disk.dm-0.in‚ 426.64453125 disk.drbd2165.in‚0 disk.sda.in‚0) proc.memory.percent‚17.21508502960205 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.6829592e+07& disk.nvme0n1.out‚36907.82275390625 disk.sda3.out‚ 338.81640625$ memory_pe‰¸–Ìrcent‚97.30665683746338# proc.memory.rssMB‚ 2820.51953125 disk.sda2.out‚0$ disk./.usageGB‚548.7621383666992 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 441.27734375) disk./.usagePercent‚36.82788754789184# disk.dm-0.out‚36907.82275390625 cpu‚26.040877853190004 network.recv‚ 2.398239e+06 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 338.81640625 disk.rbd1.in‚0.0625v‘Ÿ£"‚(:’ íõçÒ­¤º¢ gpu.0.gpu‚35.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰|n¤"‚(:÷ íõçÒÑÿؤ disk.sda1.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 427.47265625 disk.sda.out‚ 339.19140625$ memory_percent‚97.30288982391357 cpu‚26.040372566378387 network.sent‚ 1.6834595e+07$ disk./.usageGB‚548.7670860290527% disk.nvme0n1.out‚36919.6064453125 disk.sda3.in‚0! disk.rbd1.out‚106.83251953125 network.recv‚ 2.401875e+06 disk.rbd0.out‚0 disk.rbd1.in‚0.0625) proc.memory.percent‚17.21508502960205 proc.cpu.threads‚17) disk./.usagePercent‚36.82821958966375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 427.47265625 disk.dm-0.in‚ 427.47265625( proc.memory.availableMB‚ 441.89453125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚36919.6064453125 disk.sda2.out‚0 disk.dm-1.out‚ 339.19140625" disk.dm-0.out‚36919.6064453125# proc.memory.rssMB‚ 2820.51953125 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 339.19140625#Os2¥"- _wandbruntime‚9031  _runtime‚9031Sµlʦ"Ä ' lr-AdamW/pg1‚0.00022122993199785582 ' lr-AdamW/pg2‚0.00022122993199785582 trainer/global_step‚10599 % _timestamp‚1.7842818392132368e+09 _runtime‚9031.340550228 _step‚870‚&†Ï§"É _step‚870 ' lr-AdamW/pg1‚0.00022122993199785582 ' lr-AdamW/pg2‚0.00022122993199785582 trainer/global_step‚10599 % _timestamp‚1.7842818392132368e+09 _runtime‚9031.340550228æ…× 2¨"- _wandbruntime‚9031  _runtime‚9031u‰0ú'©"ôO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11209442084352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22826623 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98179033 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1622168571697664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11192339271007 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366217664207872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11335227041792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84040378 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354882515257344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11191998721117 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182209432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11209100745216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11335148950528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1613865988496735 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1602673632186368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22826358 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1610959487991808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1613865630907485 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1610959470952448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1610959470952448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11209100745216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1622168571697664 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11209442084352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41395024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354882533869568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 $ _timestamp‚1.784281839213955e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84040378 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1622168930076160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 trainer/global_step‚10599 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98179033 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 9031.34069938 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84040376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1622168571697664 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11209442084352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41395024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354882533869568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 $ _timestamp‚1.784281839213955e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84040378 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1622168930076160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 trainer/global_step‚10599 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98179033 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 9031.34069938 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84040376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22826623 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98179033 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1622168571697664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11192339271007 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366217664207872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11335227041792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84040378 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354882515257344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11191998721117 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182209432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11209100745216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11335148950528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1613865988496735 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1602673632186368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22826358 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1610959487991808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1613865630907485 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1610959470952448çñ šG2«"- _wandbruntime‚9031  _runtime‚90316hkê &¬"‚N YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11210113372160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22828393 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1602820142858240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1622317183739904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354915962628096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182226116 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84048008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41398801 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1622316825354240 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1622317183739904 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1611106711982080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1611106729021440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1614013152633079 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182236109 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11210113372160 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1611106729021440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182226116 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182236109 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84048010 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41398536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98178108 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1611106711982080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366252216710656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11210454718464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11193009774839 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366252096945664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11336254082560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10599 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18570143 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98188099 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚9031.951700774 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98178108 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354915944015872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84048010 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22828655 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11336152929792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1602820159897600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11210454718464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1614013510222441 RBDeviceStatsMonitor.on_ttë®èrain_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84048008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11193350324841ºé5³ '­"‡N E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41398801 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1622316825354240 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1622317183739904 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1611106711982080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1611106729021440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1614013152633079 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182236109 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11210113372160 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1611106729021440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182226116 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182236109 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84048010 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41398536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98178108 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1611106711982080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366252216710656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11210454718464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11193009774839 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366252096945664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11336254082560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10599 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18570143 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98188099 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚9031.951700774 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98178108 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354915944015872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84048010 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22828655 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11336152929792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1602820159897600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11210454718464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1614013510222441 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84048008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11193350324841 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11210113372160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22828393 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1602820142858240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1622317183739904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354915962628096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182226116 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84048008 Ee2µ"- _wandbruntime‚9071  _runtime‚9071&'˜¾É¶"à _runtime‚9071.129443329 _step‚874 ' lr-AdamW/pg1‚0.00021934026475112044 ' lr-AdamW/pg2‚0.00021934026475112044 trainer/global_step‚10649 $ _timestamp‚1.784281879003826e+09b­þη"È _step‚874 ' lr-AdamW/pg1‚0.00021934026475112044 ' lr-AdamW/pg2‚0.00021934026475112044 trainer/global_step‚10649 $ _timestamp‚1.784281879003826e+09 _runtime‚9071.129443329êGÊ)ã2¸"- _wandbruntime‚9071  _runtime‚9071þný'¹"÷O =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11242955757915 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367940127684096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚10649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1618321539473408 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1618321522434048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11259796247040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98623508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1618321539473408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84421978 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98623508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629581318681088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11386267271680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98633485 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚9071.130665806 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1629581677059584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183055463 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98633485 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11260137586176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚356553971795968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84421976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11259796247040 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18655651 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22929287 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84421976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84421978 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1629581677059584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7842818790048633e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367940239067648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1618321522434048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183045484 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18655648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1629581318681088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11260137586176 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1621241780987993 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1609999182819328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1621242138577243éÌÀ–Dº"üO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98633485 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11260137586176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚356553971795968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84421976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11259796247040 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18655651 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22929287 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84421976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84421978 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1629581677059584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7842818790048633e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367940239067648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1618321522434048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183045484 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18655648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1629581318681088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11260137586176 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1621241780987993 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1609999182819328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1621242138577243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11242955757915 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367940127684096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚10649 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1618321539473408 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1618321522434048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11259796247040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98623508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1618321539473408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84421978 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98623508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629581318681088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚113862672 ¼ß€¾ 71680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98633485 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚9071.130665806 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1629581677059584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183055463 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ë3½°©2»"- _wandbruntime‚9071  _runtime‚9071ŠüÑ7'¼"€N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1621389302713587 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183072161 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84429608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1629729572337664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11261150220288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41588683 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11260808873984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183062168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22931557 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 _step‚876 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11387188484096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11243966811749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367974688094208 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1610145693491200 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1610145676451840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84429610 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629729930723328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10649 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11243626261747 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183072161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98642551 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚9071.742728185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18657396 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183062168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98642551 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367974570426368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784281879616895e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11261150220288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362£ • '½"…N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11243966811749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367974688094208 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1610145693491200 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1610145676451840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84429610 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629729930723328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚10649 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11243626261747 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183072161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98642551 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚9071.742728185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18657396 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183062168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98642551 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367974570426368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784281879616895e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11261150220288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1621389302713587 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183072161 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84429608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1629729572337664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11261150220288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41588683 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11260808873984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183062168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22931557 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 _step‚876 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11387188484096ì¯$v2¾"- _wandbruntime‚9071  _runtime‚9071aò ¿"† % _timestamp‚1.7842818796201837e+09 %loss/total_loss‚7.826558935729603 epoch‚26 /loss/FRt1/detection_loss‚0.5278982352207165 1loss/SRqqt1/detection_loss‚0.8960220066364855 0loss/FRt1/assignment_loss‚1.4949532922614237 3loss/SRqqt2/assignment_loss‚0.49412816312463714 1loss/SRqqt2/detection_loss‚0.6294589491013771 _runtime‚9071.745523221 _step‚877 0loss/FRt2/assignment_loss‚0.8423275343054777 3loss/SRqqt1/assignment_loss‚0.43595128592056365 2loss/FBt1/assignment_loss‚0.031308077829827426 /loss/FRt2/detection_loss‚1.2137675890540844 0loss/FBt1/detection_loss‚0.49908659547749046 .loss/FBt2/detection_loss‚0.720512667967176 trainer/global_step‚10649 1loss/FBt2/assignment_loss‚0.04114453883034292@ƒöÀ"‹ 1loss/FBt2/assignment_loss‚0.04114453883034292 % _timestamp‚1.7842818796201837e+09 %loss/total_loss‚7.826558935729603 epoch‚26 /loss/FRt1/detection_loss‚0.5278982352207165 1loss/SRqqt1/detection_loss‚0.8960220066364855 0loss/FRt1/assignment_loss‚1.4949532922614237 3loss/SRqqt2/assignment_loss‚0.49412816312463714 1loss/SRqqt2/detection_loss‚0.6294589491013771 _runtime‚9071.745523221 _step‚877 0loss/FRt2/assignment_loss‚0.8423275343054777 3loss/SRqqt1/assignment_loss‚0.43595128592056365 2loss/FBt1/assignment_loss‚0.031308077829827426 /loss/FRt2/detection_loss‚1.2137675890540844 0loss/FBt1/detection_loss‚0.49908659547749046 .loss/FBt2/detection_loss‚0.720512667967176 trainer/global_step‚10649í|ºïû’Á"‚(:‡ šöçÒþÅ”£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚245.915' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.183 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„WF¶úÂ"‚(:ï šöçҽ俦# disk./.usageGB‚548.768627166748 disk.sda2.in‚0 disk.dm-1.out‚ 339.9140625 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 441.41015625 proc.cpu.threads‚17) disk./.usagePercent‚36.82832301670758 disk.sda3.out‚ 339.9140625& disk.nvme0n1.out‚36939.39697265625 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 429.015625 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 429.015625 disk.nvme0n1.in‚ 429.015625! disk.rbd1.out‚107.02001953125 disk.rbd0.out‚0 cpu‚26.03912105777987( disk.nvme0c0n1.out‚36939.39697265625 disk.rbd1.in‚0.0625 network.sent‚ 1.6920881e+07 disk.sda.out‚ 339.9140625# disk.dm-0.out‚36939.39697265625# proc.memory.rssMB‚ 2820.51953125 disk.sda1.out‚0$ memory_percent‚97.30584621429443 disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚ 2.413572e+06 disk.sda.in‚0 disk.rbd3.in‚0) proc.memory.percent‚17.21508502960205î)UÃ"‚(:’ ©öçÒÅ¥—¢ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±8¨d÷Ä"‚(:ì ©öçÒ…µ•¦ disk.dm-1.out‚ 342.015625" disk.dm-0.out‚36961.3720703125 disk.sda3.out‚ 342.015625 disk.rbd1.in‚0.0625$ memory_percent‚97.30396270751953 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0 cpu‚26.0385935645796 network.recv‚ 2.418772e+06$ disk./.usageGB‚548.7767028808594 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 429.73046875' disk.nvme0c0n1.out‚36961.3720703125# proc.memory.rssMB‚ 2820.51953125) disk./.usagePercent‚36.82886498465757 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 342.015625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 429.73046875% proc.memory.availableMB‚ 441.71875 disk.sda1.out‚0 disk.sda.in‚0% disk.nvme0n1.out‚36961.3720703125 disk.sda2.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.21508502960205 disk.dm-0.in‚ 429.73046875 network.sent‚ 1.6970934e+07! disk.rbd1.out‚107.02001953125 disk.rbd3.out‚0 proc.cpu.threads‚17Áàò.žÅ"‚(:“ ¸öçÒõŹ¢ gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 v ýÆ"‚(:ò ¸öçÒ±ºï¤# disk.dm-0.out‚36976.85302734375 network.sent‚ 1.6975935e+07) disk./.usagePercent‚36.82893692277469 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 344.0390625( proc.memory.availableMB‚ 441.47265625# proc.memory.rssMB‚ 2820.51953125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚36976.85302734375$ memory_percent‚97.30546474456787 network.recv‚ 2.422086e+06 disk.rbd3.in‚0 disk.nvme0n1.in‚ 430.55859375 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 430.55859375 disk.sda3.in‚0& disk.nvme0n1.out‚36976.85302734375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 344.0390625) proc.memory.percent‚17.21508502960205 proc.cpu.threads‚17# disk./.usageGB‚548.777774810791! disk.rbd1.out‚107.08251953125 disk.dm-0.in‚ 430.55859375 disk.sda3.out‚ 344.0390625 cpu‚26.03801891365 disk.sda1.out‚0 disk.dm-1.in‚0äÓô“2Ç"- _wandbruntime‚9111  _runtime‚9111³eqÊÈ"Ä % _timestamp‚1.7842819193474858e+09 _runtime‚9111.473045106 _step‚878 ' lr-AdamW/pg1‚0.00021745168898711475 ' lr-AdamW/pg2‚0.00021745168898711475 trainer/global_step‚106998wN£ÏÉ"É ' lr-AdamW/pg1‚0.00021745168898711475 ' lr-AdamW/pg2‚0.00021745168898711475 trainer/global_step‚10699 % _timestamp‚1.7842819193474858e+09 _runtime‚9111.473045106 _step‚878îf'¸í2Ê"- _wandbruntime‚9111  _runtime‚9111'z6z Ë"÷O XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18742901 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11310833088000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99087937 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11310491748864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11310833088000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1636994065664512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99087937 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23032143 % _timestamp‚1.7842819193485017e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369662768864768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1617324699373568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚10699 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1625683573915648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23032404 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358225391110144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocat…‹ÿƒed‚1169 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11293231694933 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183881536 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1625683590955008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41775041 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41775305 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1628617931068501 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9111.474360149 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1625683573915648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11437270565376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11293572244823 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84803576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99077960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11437359142400 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1617324716412928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1636994065664512 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41775041 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41775305 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1628617931068501 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚9111.474360149 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1625683573915648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11437270565376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11293572244823 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84803576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99077960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11437359142400 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1617324716412928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1636994065664512 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1636994065664512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99087937 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23032143 % _timestamp‚1.7842819193485017e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369662768864768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1617324699373568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚10699 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1625683573915648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23032404 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358225391110144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11293231694933 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183881536 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1625683590955008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1637142319321088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11294242748655 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41778872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18744646 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1637142677706752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84811208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10699 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚880 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99087012 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11311504375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11438378494976 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11311845722112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183908213 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84811210 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11438281536512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18744643 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1637142677706752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84811210 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11294583298657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1628765452794095 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1617471227084800 % _timestamp‚1.7842819199657726e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11311845722112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183898220 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1617471210045440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11311504375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358258838480896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1628765810383457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23034494 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183908213 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99097003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358258819868672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚9112.091629553 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183908213 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99097003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358258819868672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚9112.091629553 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1637142319321088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11294242748655 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41778872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18744646 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1637142677706752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84811208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚10699 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚880 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99087012 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚<ýékí 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11311504375808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11438378494976 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11311845722112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183908213 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84811210 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11438281536512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18744643 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1637142677706752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84811210 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11294583298657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1628765452794095 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1617471227084800 % _timestamp‚1.7842819199657726e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11311845722112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183898220 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1617471210045440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11311504375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358258838480896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1628765810383457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23034494 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ðÒ'2Ð"- _wandbruntime‚9112  _runtime‚9112 Ý)Ñ"‰ /loss/FRt2/detection_loss‚1.1852837837009098 1loss/SRqqt1/detection_loss‚0.8224878443090121 /loss/FBt1/detection_loss‚0.3689113889032663 /loss/FBt2/detection_loss‚0.7244100180499616 trainer/global_step‚10699 1loss/SRqqt2/detection_loss‚0.6186176051615243 &loss/total_loss‚7.6625741329400725 % _timestamp‚1.7842819199676046e+09 2loss/SRqqt1/assignment_loss‚0.4681072340799927 2loss/SRqqt2/assignment_loss‚0.4748066459583753 3loss/FBt1/assignment_loss‚0.0009068395186482531 _step‚881 0loss/FRt1/assignment_loss‚1.5879417170789387 0loss/FRt2/assignment_loss‚0.8737031115185131 4loss/FBt2/assignment_loss‚0.00021835325644120428 /loss/FRt1/detection_loss‚0.5371795914044891 epoch‚26 _runtime‚9112.092942873c”‰Ò"Ž 2loss/SRqqt1/assignment_loss‚0.4681072340799927 2loss/SRqqt2/assignment_loss‚0.4748066459583753 3loss/FBt1/assignment_loss‚0.0009068395186482531 _step‚881 0loss/FRt1/assignment_loss‚1.5879417170789387 0loss/FRt2/assignment_loss‚0.8737031115185131 4loss/FBt2/assignment_loss‚0.00021835325644120428 /loss/FRt1/detection_loss‚0.5371795914044891 epoch‚26 _runtime‚9112.092942873 /loss/FRt2/detection_loss‚1.1852837837009098 1loss/SRqqt1/detection_loss‚0.8224878443090121 /loss/FBt1/detection_loss‚0.3689113889032663 /loss/FBt2/detection_loss‚0.7244100180499616 trainer/global_step‚10699 1loss/SRqqt2/detection_loss‚0.6186176051615243 &loss/total_loss‚7.6625741329400725 % _timestamp‚1.7842819199676046e+09ñÕœt2“Ó"‚(:ˆ ÇöçÒÈËî¢ gpu.0.gpu‚87.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íÁWñÔ"‚(:æ ÇöçÒÜìæ§& disk.nvme0n1.out‚36985.45556640625 disk.sda.out‚344.5625 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 431.234375 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 431.234375 disk.rbd3.in‚0 disk.dm-0.in‚ 431.234375 disk.sda1.out‚0$ memory_percent‚97.30429649353027' proc.memory.availableMB‚ 441.6640625 cpu‚26.03750284283523 network.sent‚ 1.6980123e+07) disk./.usagePercent‚36.82894537102332 disk.dm-1.out‚344.5625# disk.dm-0.out‚36985.45556640625 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚344.5625 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2820.51953125 disk.dm-1.in‚0! disk.rbd1.out‚107.08251953125( disk.nvme0c0n1.out‚36985.45556640625) proc.memory.percent‚17.21508502960205 network.recv‚ 2.424869e+06$ disk./.usageGB‚548.7779006958008€sÓ3‘Õ"‚(:† ÖöçÒùɹ¢ gpu.0.gpu‚48.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äßþMÖ"‚(:ú ÖöçÒþ“ì¤ network.sent‚ 1.7043208e+07 disk.nvme0n1.in‚ 431.94921875 disk.sda.out‚ 346.83203125 disk.sda3.out‚ 346.83203125 disk.drbd2165.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 431.94921875( disk.nvme0c0n1.out‚36997.31884765625 disk.sda.in‚0 disk.sda2.out‚0 network.recv‚ 2.429755e+06$ disk./.usageGB‚548.7746391296387 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0# disk.dm-0.out‚36997.31884765625 disk.drbd2165.out‚0 disk.dm-0.in‚ 431.94921875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 346.83203125# proc.memory.rssMB‚ 2820.51953125) proc.memory.percent‚17.21508502960205 cpu‚26.036996152933334 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚97.3059892654419* disk./.usagePercent‚36.828726484581544& disk.nvme0n1.out‚36997.31884765625 disk.dm-1.in‚0! disk.rbd1.out‚107.14501953125 disk.rbd3.out‚0( proc.memory.availableMB‚ 441.38671875Y@£‘×"‚(:† åöçÒô’ª¢ gpu.0.gpu‚89.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y–”Ø"‚(:÷ åöçÒÉÚ¦¤% disk.nvme0n1.out‚37009.3330078125 disk.rbd0.in‚0) proc.memory.percent‚17.21508502960205 disk.rbd1.in‚0.0625 network.sent‚ 1.7048238e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 network.recv‚ 2.433459e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 348.11328125! disk.rbd1.out‚107.14501953125" disk.nvme0c0n1.in‚ 432.77734375# memory_percent‚97.3059892654419 disk.sda.in‚0$ disk./.usageGB‚548.7715110778809 disk.rbd3.in‚0' disk.nvme0c0n1.out‚37009.3330078125 disk.sda.out‚ 348.11328125 disk.sda2.in‚0# proc.memory.rssMB‚ 2820.51953125 disk.sda3.in‚0" disk.dm-0.out‚37009.3330078125* disk./.usagePercent‚36.828516558403464 disk.sda3.out‚ 348.11328125 disk.drbd2165.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 441.38671875 cpu‚26.036422476304374 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.in‚ 432.77734375 disk.nvme0n1.in‚ 432.77734375ÐM I2Ù"- _wandbruntime‚9151  _runtime‚9151ˆæÊÚ"Ä ' lr-AdamW/pg1‚0.00021556432600553104 ' lr-AdamW/pg2‚0.00021556432600553104 trainer/global_step‚10749 % _timestamp‚1.7842819595896614e+09 _runtime‚9151.715374157 _step‚882Œ¯ºÏÛ"É _runtime‚9151.715374157 _step‚882 ' lr-AdamW/pg1‚0.00021556432600553104 ' lr-AdamW/pg2‚0.00021556432600553104 trainer/global_step‚10749 % _timestamp‚1.7842819595896614e+09òLÔ –2Ü"- _wandbruntime‚9151  _runtime‚9151¡R_ü'Ý"öO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11361187250688 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85185178 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1633045625397248 $ _timestamp‚1.784281959590857e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23135632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1624650250006528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99542389 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359896847648768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11361187250688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184717588 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1635994438738259 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11344188731731 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41965507 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚9151.716659584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1624650232967168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99542389 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1633045625397248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371385420290048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99532412 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11361528589824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1644407171026432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371385311003648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41965783 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1644406812647936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85185176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18830148 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1635994081149009 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11343848181841 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99532412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359896829036544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18830151 trainer/global_step‚10749 _step‚883 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1644407171026432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11361528589824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184727567 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1644406812647936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85185176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18830148 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1635994081149009 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11343848181841 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99532412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359896829036544 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18830151 trainer/global_step‚10749 _step‚883 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1644407171026432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11361528589824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184727567 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184717588 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1635994438738259 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11344188731731 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41965507 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚9151.716659584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1624650232967168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99542389óH¿™ê2ß"- _wandbruntime‚9152  _runtime‚9152/“÷gà"N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1636141602874603 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85192808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1633192883466240 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/sei—¯r "gment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184744265 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99541464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99551455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11489504235008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85192810 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1644555424690176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11345199785565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18831896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1644555066304512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41969316 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11344859235563 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85192810 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1633192866426880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚9152.328924856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11362541223936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11362199877632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1636141960463965 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1633192866426880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85192808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371419762030080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1624796743639040 trainer/global_step‚10749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184744265 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842819602030046e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1624796760678400 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11362541223936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99551455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23137689 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184734272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1644555424690176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18831893 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚419695850Wý‡ 'á"†N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1633192883466240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23137423 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _step‚884 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99541464 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11345199785565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18831896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1644555066304512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41969316 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11344859235563 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85192810 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1633192866426880 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚9152.328924856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11362541223936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11362199877632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1636141960463965 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1633192866426880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85192808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371419762030080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1624796743639040 trainer/global_step‚10749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184744265 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842819602030046e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1624796760678400 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11362541223936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99551455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23137689 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184734272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1644555424690176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18831893 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41969585 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1636141602874603 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85192808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1633192883466240 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184744265 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99541464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99551455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11489504235008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85192810 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1644555424690176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1651819559631360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373108045605376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361568266962944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11412224091648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99996841 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚10799 _step‚887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1631975783600128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11411882752512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85566776 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1640407693918208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1643370231229517 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1651819918009856 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1640407676878848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85566778 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42156332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18917401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18917398 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1651819918009856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚9191.405401623 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99986864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99996841 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚373107930027520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11394805218639 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1631975766560768 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11412224091648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185563619 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42156601 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842819992762148e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99986864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1651819559631360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start×PL&^/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1651819559631360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373108045605376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361568266962944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11412224091648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99996841 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚10799 _step‚887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1631975783600128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11411882752512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85566776 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1640407693918208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1643370231229517 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1651819918009856 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1640407676878848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85566778 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42156332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18917401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18917398 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1651819918009856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚9191.405401623 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99986864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99996841 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚373107930027520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11394805218639 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1631975766560768 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11412224091648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185563619 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42156601 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842819992762148e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99986864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1651819559631360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983÷OŸ–2ï"- _wandbruntime‚9192  _runtime‚9192u½t 'ð"ƒN D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185570324 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373142374012928 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1643517752955111 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85574410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11412895379456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185580317 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11540773152768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11540678291456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18919146 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42160134 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373142487486464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42160399 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100005907 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18919143 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11412895379456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361601714333696 % _timestamp‚1.7842819998879857e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99995916 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1643518110544473 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚9192.013867447 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185580317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361601695721472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1640554934947840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1632122294272000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1640554917908480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23241253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99995916 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11395475722471 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85574410 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11413236725760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11413236725760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11395816272473 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10799`ze'ñ"ˆN ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1640554934947840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1632122294272000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1640554917908480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23241253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99995916 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11395475722471 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85574410 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11413236725760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11413236725760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11395816272473 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185570324 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373142374012928 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1643517752955111 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85574410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11412895379456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185580317 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11540773152768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11540678291456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18919146 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42160134 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373142487486464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42160399 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100005907 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18919143 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11412895379456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361601714333696 % _timestamp‚1.7842819998879857e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99995916 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1643518110544473 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚9192.013867447 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185580317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361601695721472ø°·eå2ò"- _wandbruntime‚9192  _runtime‚9192Œ§¿ ó"‡ 0loss/FRt2/assignment_loss‚0.8236479531174247 /loss/FRt1/detection_loss‚0.4874528837451546 1loss/SRqqt2/detection_loss‚0.6188099332700149 trainer/global_step‚10799 % _timestamp‚1.7842819998914688e+09 _step‚889 0loss/FRt1/assignment_loss‚1.6180422875014218 3loss/SRqqt1/assignment_loss‚0.44235791876145086 1loss/SRqqt1/detection_loss‚0.9069191051290391 0loss/FBt1/detection_loss‚0.46775720407486077 /loss/FBt2/detection_loss‚0.8090036037826428 %loss/total_loss‚7.930115642998546 epoch‚26 1loss/FBt1/assignment_loss‚0.04362777081758977 3loss/FBt2/assignment_loss‚0.0017839510455756992 /loss/FRt2/detection_loss‚1.2723619976865226 2loss/SRqqt2/assignment_loss‚0.4383510340668486 _runtime‚9192.016829347þúSèàô"Œ 2loss/SRqqt2/assignment_loss‚0.4383510340668486 _runtime‚9192.016829347 0loss/FRt2/assignment_loss‚0.8236479531174247 /loss/FRt1/detection_loss‚0.4874528837451546 1loss/SRqqt2/detection_loss‚0.618öIÆô28099332700149 trainer/global_step‚10799 % _timestamp‚1.7842819998914688e+09 _step‚889 0loss/FRt1/assignment_loss‚1.6180422875014218 3loss/SRqqt1/assignment_loss‚0.44235791876145086 1loss/SRqqt1/detection_loss‚0.9069191051290391 0loss/FBt1/detection_loss‚0.46775720407486077 /loss/FBt2/detection_loss‚0.8090036037826428 %loss/total_loss‚7.930115642998546 epoch‚26 1loss/FBt1/assignment_loss‚0.04362777081758977 3loss/FBt2/assignment_loss‚0.0017839510455756992 /loss/FRt2/detection_loss‚1.2723619976865226ùE%Íy“õ"‚(:ˆ ’÷çÒìå£ gpu.0.gpu‚66.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~x³Þûö"‚(:ð ’÷çÒ†ïϦ! disk.nvme0c0n1.in‚ 434.3203125 disk.sda.in‚0 disk.dm-0.in‚ 434.3203125 disk.dm-1.in‚0! disk.rbd1.out‚107.33251953125 disk.nvme0n1.in‚ 434.3203125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 network.recv‚ 2.445385e+06) disk./.usagePercent‚36.82831533648156% disk.nvme0n1.out‚37036.3779296875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚97.30470180511475' disk.nvme0c0n1.out‚37036.3779296875 disk.dm-1.out‚ 350.8671875" disk.dm-0.out‚37036.3779296875# proc.memory.rssMB‚ 2820.51953125) proc.memory.percent‚17.21508502960205 network.sent‚ 1.7134472e+07$ disk./.usageGB‚548.7685127258301 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 350.8671875 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.sda3.out‚ 350.8671875( proc.memory.availableMB‚ 441.59765625 cpu‚26.03596088943557 proc.cpu.threads‚17dNºÏ2÷"- _wandbruntime‚9209  _runtime‚9209Ç5 Å=ø"¿{ :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30040591955184937 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884964227676392 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6485887765884399 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23664499819278717 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6931456327438354 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6970687508583069 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8372449278831482 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2495144158601761 _step‚890 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4592002332210541 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583134651184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4642654061317444 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05421615391969681 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9774606823921204 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25599896907806396 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7310469150543213 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.018040023744106293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7869628071784973 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.024610362946987152 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6941946744918823 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24503985047340393 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18917515873908997 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5416666865348816 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958450794219971 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7121268510818481 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6725271940231323 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995637059211731 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6587607264518738 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.018268808722496033 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7080945372581482 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7648361921310425 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.024610362946987152 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7584824562072754 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6679762601852417 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.517686665058136 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20392468571662903 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964407682418823 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 (particle/accuracy‚0.8452225923538208 'particle/f_score‚0.6319382786750793 jet/accuracy_1_of_5‚1 jet/accuracy_3_of_5‚1 _runtime‚9209.280375004 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2344897985458374 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958730697632 )loss/val_total_loss‚7.994881327490374 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18545974791049957 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30335161089897156 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6464663147926331 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6596544981002808 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7201712727546692 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8486964702606201 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24430304765701294 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6386429667472839 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7501368522644043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_1‚0.3387366831302643 7validation_average_jet_accuracy‚0.37896671891212463 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12260965257883072 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4911893308162689 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6358122825622559 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3743484318256378 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0291900634765625 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.46875 jet/accuracy_2_of_5‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1508333384990692 ,jet/accuracy_4_of_4‚0.006769374944269657 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23635876178741455 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8937270045280457 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4305555522441864 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_2‚0.7914149761199951 +particle/specificity‚0.9071471095085144 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2012263536453247 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3902423083782196 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9083506464958191 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.204109787940979 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5773898363113403 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04564249888062477 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.749098539352417 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25640833377838135 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8656625151634216 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7196961641311646 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_2_of_4‚0.7897223234176636 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8029887676239014 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02260705456137657 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 trainer/global_step‚10799 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884825944900513 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 +jet/accuracy_3_of_4‚0.07262129336595535 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21481822431087494 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035275381058454514 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9955486059188843 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985058903694153 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6622136235237122 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5988423228263855 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.740783154964447 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33390480279922485 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9924123883247375 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8656625151634216 particle/accuracy_3_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.020763413980603218 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6498446464538574 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.745445966720581 )jet/accuracy_1_of_3‚0.921777606010437 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2322205752134323 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6855279207229614 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7114735841751099 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7071806788444519 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2552270293235779 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12719900906085968 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7497017979621887 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7081444263458252 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19477732479572296 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 +jet/accuracy_2_of_2‚0.15247848629951477 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06801064312458038 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930187463760376 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11005902290344238 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48882433772087097 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6622136235237122 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949442744255066 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9836840033531189 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7427619695663452 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8047360181808472 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02093307487666607 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7175020575523376 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9259598255157471 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +particle/sensitivity‚0.6187856793403625 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38875553011894226 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5329886674880981 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2743982672691345 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20248940587043762 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5676694512367249 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7813740968704224 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 epoch‚26 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 particle/accuracy_1_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24503985047340393 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6479842066764832 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969876408576965 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04167651757597923 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4642857015132904 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7303099632263184 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21614143252372742 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.218215212225914 % _timestamp‚1.7842820171540275e+09 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7530810832977295 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6228245496749878 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1400637924671173 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.602167546749115 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978471398353577 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9918419718742371 *jet/accuracy_2_of_3‚0.5659521818161011 /particle/accuracy_4_of_4‚0.9474366307258606 particle/accuracy_5_of_5‚16}5Á:ù"Ä{ @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6941946744918823 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24503985047340393 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18917515873908997 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5416666865348816 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958450794219971 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7121268510818481 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6725271940231323 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995637059211731 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6587607264518738 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.018268808722496033 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7080945372581482 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7648361921310425 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.024610362946987152 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7584824562072754 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6679762601852417 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.517686665058136 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20392468571662903 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964407682418823 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 (particle/accuracy‚0.8452225923538208 'particle/f_score‚0.6319382786750793 jet/accuracy_1_of_5‚1 jet/accuracy_3_of_5‚1 _runtime‚9209.280375004 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2344897985458374 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958730697632 )loss/val_total_loss‚7.994881327490374 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18545974791049957 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30335161089897156 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6464663147926331 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6596544981002808 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7201712727546692 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8486964702606201 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24430304765701294 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6386429667472839 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7501368522644043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_1‚0.3387366831302643 7validation_average_jet_accuracy‚0.37896671891212463 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12260965257883072 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4911893308162689 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6358122825622559 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3743484318256378 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0291900634765625 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.46875 jet/accuracy_2_of_5‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1508333384990692 ,jet/accuracy_4_of_4‚0.006769374944269657 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.23635876178741455 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8937270045280457 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4305555522441864 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_2‚0.7914149761199951 +particle/specificity‚0.9071471095085144 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2012263536453247 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3902423083782196 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9083506464958191 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.204109787940979 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5773898363113403 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04564249888062477 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.749098539352417 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25640833377838135 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8656625151634216 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7196961641311646 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_2_of_4‚0.7897223234176636 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8029887676239014 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02260705456137657 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 trainer/global_step‚10799 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884825944900513 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 +jet/accuracy_3_of_4‚0.07262129336595535 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21481822431087494 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035275381058454514 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9955486059188843 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985058903694153 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6622136235237122 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5988423228263855 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.740783154964447 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33390480279922485 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9924123883247375 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8656625151634216 particle/accuracy_3_of_5‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.020763413980603218 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6498446464538574 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.745445966720581 )jet/accuracy_1_of_3‚0.921777606010437 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2322205752134323 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6855279207229614 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7114735841751099 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7071806788444519 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2552270293235779 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12719900906085968 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7497017979621887 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7081444263458252 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19477732479572296 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 +jet/accuracy_2_of_2‚0.15247848629951477 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06801064312458038 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930187463760376 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11005902290344238 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48882433772087097 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6622136235237122 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949442744255066 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9836840033531189 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7427619695663452 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8047360181808472 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02093307487666607 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7175020575523376 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9259598255157471 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +particle/sensitivity‚0.6187856793403625 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38875553011894226 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5329886674880981 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2743982672691345 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20248940587043762 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5676694512367249 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7813740968704224 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 epoch‚26 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_5‚1 particle/accuracy_1_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24503985047340393 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6479842066764832 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969876408576965 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04167651757597923 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4642857015132904 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7303099632263184 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21614143252372742 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.218215212225914 % _timestamp‚1.7842820171540275e+09 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7530810832977295 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6228245496749878 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1400637924671173 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.602167546749115 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978471398353577 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9918419718742371 *jet/accuracy_2_of_3‚0.5659521818161011 /particle/accuracy_4_of_4‚0.9474366307258606 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30040591955184937 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884964227676392 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6485887765884399 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23664499819278717 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6931456327438354 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6970687508583069 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8372449278831482 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2495144158601761 _step‚890 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt1FBt/event_purity‚™¥U«0.4592002332210541 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583134651184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4642654061317444 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05421615391969681 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9774606823921204 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25599896907806396 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7310469150543213 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.018040023744106293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7869628071784973 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.024610362946987152úQôí“ú"‚(:ˆ ¡÷çÒßÿ¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚246.157' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.2314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08â!÷û"‚(:ì ¡÷çÒ³ø¼¦ disk.rbd1.in‚0.0625$ memory_percent‚97.31242656707764 cpu‚26.041710857114577 network.sent‚ 1.7197336e+07 network.recv‚ 2.45089e+06* disk./.usagePercent‚36.828345801378134 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda.out‚ 351.109375# proc.memory.rssMB‚ 2821.83984375 disk.nvme0n1.in‚ 435.03515625 disk.sda2.in‚0 disk.sda3.out‚ 351.109375 disk.dm-1.in‚0 disk.dm-0.in‚ 435.03515625 disk.dm-0.out‚37045.87890625( proc.memory.availableMB‚ 440.33203125* proc.memory.percent‚17.223143577575684$ disk./.usageGB‚548.7689666748047" disk.nvme0c0n1.in‚ 435.03515625 disk.sda2.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 351.109375! disk.rbd1.out‚107.33251953125 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚37045.87890625# disk.nvme0n1.out‚37045.87890625÷.Ä|ü"‚(:’ °÷ç݄҃¤ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚262.071' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZãRý"‚(:õ °÷çÒ°ãï¤ network.recv‚ 2.455683e+06! disk.rbd1.out‚107.39501953125* proc.memory.percent‚17.223143577575684 disk.rbd3.out‚0 network.sent‚ 1.7242133e+07 disk.nvme0n1.in‚ 435.86328125 disk.sda.in‚0% proc.memory.availableMB‚ 440.34375% disk.nvme0n1.out‚37053.8291015625 disk.sda3.out‚ 351.30859375# memory_percent‚97.3123550415039 cpu‚26.041194264769143 disk.sda3.in‚0 disk.dm-1.out‚ 351.30859375" disk.dm-0.out‚37053.8291015625 disk.dm-1.in‚0 disk.dm-0.in‚ 435.86328125" disk.nvme0c0n1.in‚ 435.86328125* disk./.usagePercent‚36.828340681227445$ disk./.usageGB‚548.7688903808594 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚37053.8291015625 disk.sda.out‚ 351.30859375 disk.sda2.out‚0# proc.memory.rssMB‚ 2821.83984375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0y½æmžþ"‚(:“ ¿÷çÒ“ÊÏ¢ gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚200.388' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.077600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Ò¹ýÿ"‚(:ò ¿÷çÒõ÷˜¦ disk.nvme0n1.in‚ 436.578125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0& disk.nvme0n1.out‚37063.79345703125 disk.rbd0.out‚0$ memory_percent‚97.31385707855225# proc.memory.rssMB‚ 2821.83984375 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 2.459171e+06 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 440.09765625 cpu‚26.040662358277743 disk.dm-1.in‚0 disk.dm-0.in‚ 436.578125( disk.nvme0c0n1.out‚37063.79345703125 proc.cpu.threads‚17( disk./.usagePercent‚36.8283470814158# disk./.usageGB‚548.768985748291 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 1.724717e+07 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 436.578125 disk.sda.out‚ 351.51953125* proc.memory.percent‚17.223143577575684 disk.dm-1.out‚ 351.51953125! disk.rbd1.out‚107.39501953125# disk.dm-0.out‚37063.79345703125 disk.sda3.out‚ 351.51953125Ý°Iñ2€#- _wandbruntime‚9248  _runtime‚92483–¼Ê#Ä % _timestamp‚1.7842820563207932e+09 _runtime‚9248.446435382 _step‚891 ' lr-AdamW/pg1‚0.00021179372319113878 ' lr-AdamW/pg2‚0.00021179372319113878 trainer/global_step‚10849òºæÏ‚#É _step‚891 ' lr-AdamW/pg1‚0.00021179372319113878 ' lr-AdamW/pg2‚0.00021179372319113878 trainer/global_step‚10849 % _timestamp‚1.7842820563207932e+09 _runtime‚9248.446435382û©Çл2ƒ#- _wandbruntime‚9248  _runtime‚9248Au^2(„#ûO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19007168 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1649699295010816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1649699295010816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42380345 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11609100902912 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842820563219063e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1661179964849152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11462800880886 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19007171 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374864736504320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11609018617344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚9248.447707123 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86069044 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363255616989184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11480652798976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186590659 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23372885 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11463141430776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚10849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186580680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86069042 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1641209963368448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100521615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100511638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23373174 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363255635601408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1661179606470656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86069042 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11480311459840 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1649699312050176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1641209980407808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100521615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1661179964849152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100511638 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374864635606528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1661179606470656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86069044 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169sLàË(…#€P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42380053 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1661179606470656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86069042 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11480311459840 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1649699312050176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1641209980407808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100521615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1661179964849152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100511638 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374864635606528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1661179606470656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚892 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86069044 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19007168 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1649699295010816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1649699295010816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42380345 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11609100902912 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842820563219063e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1661179964849152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11462800880886 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19007171 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374864736504320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11609018617344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚9248.447707123 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86069044 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363255616989184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11480652798976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186590659 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23372885 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11463141430776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚10849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186580680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86069042 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1641209963368448 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100521615 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100511638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23373174 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363255635601408 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11481324086784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11481324086784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86076674 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11481665433088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11481665433088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186597364 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374899198906368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100530681 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23374952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1652820643564290 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100520690 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19008913 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374899102210048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186597364 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363289064359936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186607357 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86076676 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42384116 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86076674 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100520690 _step‚893 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42383865 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Õ5p/'ˆ#‰N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11610134546432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11463811934608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186607357 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1649846536040448 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1649846553079808 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1661328218512896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 % _timestamp‚1.7842820569345398e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100530681 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1652820285974928 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1641356474040320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1649846553079808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚9249.060484892 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1661327860127232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186607357 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86076676 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42384116 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86076674 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100520690 _step‚893 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42383865 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1649846536040448 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11481324086784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11481324086784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86076674 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11481665433088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11481665433088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186597364 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374899198906368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100530681 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23374952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1652820643564290 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100520690 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19008913 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374899102210048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186597364 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚363289064359936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11531006961664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11531006961664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187426711 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1657061363531776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364927054915584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86450644 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1668592711832576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1668592353454080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1648535514001408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11531348300800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19094418 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376587553915904 _step‚896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1657061346492416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11531348300800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42571168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11660499000320 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100966090 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1668592711832576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11660579188736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187416732 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86450644 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438Ìvûs —#ÿO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100976067 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100966090 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1648535496962048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86450642 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚10899 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1660048914329842 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1668592353454080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23476750 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364927073527808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11531006961664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11531006961664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187426711 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1657061363531776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364927054915584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86450644 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1668592711832576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1668592353454080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1648535514001408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11531348300800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19094418 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376587553915904 _step‚896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1657061346492416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11531348300800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42571168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11660499000320 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100966090 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1668592711832576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11660579188736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187416732 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86450644 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438€¯HÏA2˜#- _wandbruntime‚9289  _runtime‚9289Ö>/ '™#…N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187443409 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1660196436055436 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1648682024673280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100975142 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11532360934912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19096166 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23479065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187433416 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1657208604561408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11661606131200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11514768971518 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1657208587522048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187433416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100985133 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376621994944000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19096163 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚897 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187443409 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11532019588608 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1657208604561408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1668740965496320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11532019588608 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364960502286336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11532360934912 _runtime‚9289.180478571 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1668740607110656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376622108417536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1657208587522048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42575231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842820970545676e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86458274 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1668740965496320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86458274 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42574962 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11514428421516 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚10899 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364960483674112 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1660196793644798 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86458276 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100985133 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1648682007633920ú ’Â'š#ŠN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86458276 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100985133 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1648682007633920 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187443409 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1660196436055436 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1648682024673280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100975142 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11532360934912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19096166 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23479065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187433416 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1657208604561408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11661606131200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11514768971518 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1657208587522048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187433416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100985133 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376621994944000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19096163 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚897 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187443409 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11532019588608 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1657208604561408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1668740965496320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11532019588608 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364960502286336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11532360934912 _runtime‚9289.180478571 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1668740607110656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376622108417536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1657208587522048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42575231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842820970545676e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86458274 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1668740965496320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86458274 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42574962 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11514428421516 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚10899 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364960483674112 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1660196793644798Û?V 2›#- _wandbruntime‚9289  _runtime‚9289)6g4 œ#† epoch‚27 /loss/FRt2/assignment_loss‚0.878032775505169 2loss/SRqqt1/assignment_loss‚0.4509853423326357 2loss/FBt1/assignment_loss‚0.014703473536497537 /loss/FRt2/detection_loss‚1.1803188311191997 trainer/global_step‚10899 % _timestamp‚1.7842820970579278e+09 3loss/SRqqt2/assignment_loss‚0.46691250591307587 /loss/FRt1/detection_loss‚0.5345824245383021 1loss/SRqqt1/detection_loss‚0.8409706821568155 1loss/SRqqt2/detection_loss‚0.6295129524226805 /loss/FBt1/detection_loss‚0.4044022761134905 _runtime‚9289.183273577 0loss/FRt1/assignment_loss‚1.4949558376583956 2loss/FBt2/assignment_loss‚0.001893448631512268 /loss/FBt2/detection_loss‚0.5645096290694631 &loss/total_loss‚7.4617801789972376 _step‚898@û3½#‹ /loss/FBt2/detection_loss‚0.5645096290694631 &loss/total_loss‚7.4617801789972376 _step‚898 epoch‚27 /loss/FRt2/assignment_loss‚0.878032775505169 2loss/SRqqt1/assignment_loss‚0.4509853423326357 2loss/FBt1/assignment_loss‚0.014703473536497537 /loss/FRt2/detection_loss‚1.1803188311191997 trainer/global_step‚10899 % _timestamp‚1.7842820970579278e+09 3loss/SRqqt2/assignment_loss‚0.46691250591307587 /loss/FRt1/detection_loss‚0.5345824245383021 1loss/SRqqt1/detection_loss‚0.8409706821568155 1loss/SRqqt2/detection_loss‚0.6295129524226805 /loss/FBt1/detection_loss‚0.4044022761134905 _runtime‚9289.183273577 0loss/FRt1/assignment_loss‚1.4949558376583956 2loss/FBt2/assignment_loss‚0.001893448631512268‚SÆ8”“ž#‚(:ˆ û÷çÒ¶ö§¥ gpu.0.gpu‚37.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë)ØyŸ#‚(:ú û÷çÒßöݦ disk.nvme0n1.in‚ 438.94921875 disk.rbd0.in‚0 disk.dm-1.out‚ 359.91015625" disk.nvme0c0n1.in‚ 438.94921875' proc.memory.availableMB‚ 439.8984375* proc.memory.percent‚17.223143577575684 cpu‚26.038686125514758 network.sent‚ 1.7338521e+07 network.recv‚ 2.474912e+06 disk.sda.out‚ 359.91015625 disk.sda2.in‚0 disk.sda3.out‚ 359.91015625$ memory_percent‚97.31507301330566& disk.nvme0n1.out‚37132.36962890625 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0! disk.rbd1.out‚107.52001953125# disk.dm-0.out‚37132.36962890625# proc.memory.rssMB‚ 2821.83984375) disk./.usagePercent‚36.82871189215209 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.in‚0.0625( disk.nvme0c0n1.out‚37132.36962890625 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 438.94921875$ disk./.usageGB‚548.7744216918945ÌÅ=ž #‚(:“ ŠøçÒþ½º¢ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˼  ¡#‚(:õ ŠøçÒáüŠ¥ disk.sda2.in‚0 disk.dm-0.in‚ 439.6640625$ memory_percent‚97.31645584106445& proc.memory.availableMB‚ 439.671875 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 360.46484375# proc.memory.rssMB‚ 2821.83984375 cpu‚26.038206351959502( disk./.usagePercent‚36.8284412921884 disk.rbd0.out‚0 network.recv‚ 2.480104e+06 disk.drbd2165.out‚0 disk.sda.in‚0* proc.memory.percent‚17.223143577575684 proc.cpu.threads‚17 network.sent‚ 1.7388551e+07 disk.nvme0n1.in‚ 439.6640625& disk.nvme0n1.out‚37139.78076171875 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚37139.78076171875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0$ disk./.usageGB‚548.7703895568848! disk.nvme0c0n1.in‚ 439.6640625 disk.sda3.in‚0 disk.dm-1.out‚ 360.46484375 disk.sda3.out‚ 360.46484375! disk.rbd1.out‚107.52001953125# disk.dm-0.out‚37139.78076171875 disk.drbd2165.in‚0¶ú h2¢#- _wandbruntime‚9328  _runtime‚9328!oqÈ£# % _timestamp‚1.7842821360991821e+09 _runtime‚ 9328.2248004 _step‚899 ' lr-AdamW/pg1‚0.00020802942500748017 ' lr-AdamW/pg2‚0.00020802942500748017 trainer/global_step‚10949±ˆy@¤#Ç % _timestamp‚1.7842821360991821e+09 _runtime‚ 9328‡øÒë.2248004 _step‚899 ' lr-AdamW/pg1‚0.00020802942500748017 ' lr-AdamW/pg2‚0.00020802942500748017 trainer/global_step‚10949ƒM;ar2¥#- _wandbruntime‚9328  _runtime‚93285Â,(¦#úO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101420542 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19181671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366598492841984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188262763 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1655861047595008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1664423397974016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11564033854702 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1667425064410350 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1664423397974016 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42761197 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86832242 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101420542 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101430519 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1676005458816000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86832242 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378310459373568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86832244 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1676005100437504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19181668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378310358475776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1676005458816000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101430519 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.784282136100374e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10949 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1676005100437504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23579803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11581702463488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1655861030555648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚366598511454208 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1667425421999600 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088÷ÀÒ­(§#ÿO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚9328.226134549 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11582043802624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11582043802624 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.784282136100374e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10949 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1676005100437504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23579803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11581702463488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1655861030555648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚366598511454208 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1667425421999600 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101420542 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19181671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366598492841984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188262763 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1655861047595008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1664423397974016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11564033854702 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1667425064410350 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1664423397974016 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42761197 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86832242 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101420542 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101430519 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1676005458816000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86832242 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378310459373568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86832244 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1676005100437504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19181668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378310358475776 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1676005458816000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101430519 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42765007 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101429594 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1667572943725306 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11565385458426 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11565044908424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 9328.83690395 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚366631940212736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1664570639003648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188269468 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11582715090432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101429594 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101439585 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188279461 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23581594 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101439585 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1676153354094080 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 9328.83690395 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚366631940212736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_poolEׄ>.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1664570639003648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188269468 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11582715090432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101429594 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101439585 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188279461 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23581594 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101439585 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1676153354094080 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42765007 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101429594 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072…»¸´F2«#- _wandbruntime‚9328  _runtime‚9328ù_\H ¬#‡ /loss/FRt2/detection_loss‚1.2302263162270077 /loss/FBt2/detection_loss‚0.8332981224482258 %loss/total_loss‚7.437301827156006 /loss/FRt1/assignment_loss‚1.541678945509206 2loss/FBt1/assignment_loss‚0.006439549077479444 1loss/SRqqt1/detection_loss‚0.8229904193996448 1loss/SRqqt2/detection_loss‚0.6201055502921493 epoch‚27 trainer/global_step‚10949 % _timestamp‚1.7842821367136347e+09 _runtime‚9328.838972375 0loss/FBt1/detection_loss‚0.32041086360547727 /loss/FRt2/assignment_loss‚0.754574824138958 _step‚902 3loss/SRqqt1/assignment_loss‚0.34765947326131974 3loss/SRqqt2/assignment_loss‚0.47644699287903436 3loss/FBt2/assignment_loss‚5.106040980488539e-06 /loss/FRt1/detection_loss‚0.4834656642765226hjÕ¿­#Œ % _timestamp‚1.7842821367136347e+09 _runtime‚9328.838972375 0loss/FBt1/detection_loss‚0.32041086360547727 /loss/FRt2/assignment_loss‚0.754574824138958 _step‚902 3loss/SRqqt1/assignment_loss‚0.34765947326131974 3loss/SRqqt2/assignment_loss‚0.47644699287903436 3loss/FBt2/assignment_loss‚5.106040980488539e-06 /loss/FRt1/detection_loss‚0.4834656642765226 /loss/FRt2/detection_loss‚1.2302263162270077 /loss/FBt2/detection_loss‚0.8332981224482258 %loss/total_loss‚7.437301827156006 /loss/FRt1/assignment_loss‚1.541678945509206 2loss/FBt1/assignment_loss‚0.006439549077479444 1loss/SRqqt1/detection_loss‚0.8229904193996448 1loss/SRqqt2/detection_loss‚0.6201055502921493 epoch‚27 trainer/global_step‚10949†a>ðÀ‘®#‚(:† ™øçÒàƘ¢ gpu.0.gpu‚50.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚251.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ã€ô¯#‚(:é ™øçÒûÿ“¤! disk.rbd1.out‚107.58251953125 network.recv‚ 2.483428e+06 disk.sda1.in‚0 disk.dm-1.out‚361.1875$ memory_percent‚97.31683731079102) disk./.usagePercent‚36.82845076446717 disk.drbd2165.in‚0 disk.dm-0.in‚ 439.6640625 disk.rbd3.in‚0* proc.memory.percent‚17.223143577575684 cpu‚26.037697287062475& disk.nvme0n1.out‚37147.41845703125 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2821.83984375 network.sent‚ 1.739356e+07$ disk./.usageGB‚548.7705307006836 disk.nvme0n1.in‚ 439.6640625 disk.rbd0.in‚0 disk.sda3.out‚361.1875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚361.1875 disk.sda2.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚37147.41845703125& proc.memory.availableMB‚ 439.609375! disk.nvme0c0n1.in‚ 439.6640625 disk.rbd1.in‚0.0625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚37147.41845703125 disk.sda.in‚0ÀÊÞO’°#‚(:‡ ¨øçÒ°ÚÛ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­èŠò±#‚(:÷ ¨øçÒêÉ¥ disk.sda1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚37159.37451171875 disk.sda.out‚ 361.49609375 disk.sda3.in‚0( proc.memory.availableMB‚ 439.76171875# proc.memory.rssMB‚ 2821.83984375 network.recv‚ 2.488609e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚37159.37451171875 network.sent‚ 1.7456668e+07) disk./.usagePercent‚36.82866222669045 disk.nvme0n1.in‚ 440.4921875 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 361.49609375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 361.49609375! disk.rbd1.out‚107.58251953125 disk.dm-0.in‚ 440.4921875$ memory_percent‚97.31590747833252* proc.memory.percent‚17.223143577575684# disk./.usageGB‚548.773681640625! disk.nvme0c0n1.in‚ 440.4921875& disk.nvme0n1.out‚37159.37451171875 disk.rbd0.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0.0625 cpu‚26.037149667853733i5Ä¥²#‚(:’ ·øçÒ’¯á¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ¼“þ³#‚(:ó ·øçÒ£æö¥ disk.sda2.out‚0# proc.memory.rssMB‚ 2821.83984375* disk./.usagePercent‚36.828829911625384 cpu‚26.036663113309466 network.recv‚ 2.49211e+06! disk.rbd1.out‚107.64501953125" disk.dm-0.out‚37181.8896484375 disk.sda2.in‚0 network.sent‚ 1.7461662e+07 disk.sda1.in‚0 disk.drbd2165.out‚0# memory_percent‚97.3165512084961 proc.cpu.threads‚17% disk.nvme0n1.out‚37181.8896484375% proc.memory.availableMB‚ 439.65625 disk.sda3.out‚ 363.85546875 disk.rbd3.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚37181.8896484375 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 363.85546875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 441.20703125 disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 441.20703125 disk.drbd2165.in‚0* proc.memory.percent‚17.223143577575684# disk./.usageGB‚548.776180267334 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 441.20703125 disk.sda.out‚ 363.85546875œ 2´#- _wandbruntime‚9368  _runtime‚93681†±Éµ#à % _timestamp‚1.7842821760606587e+09 _runtime‚ 9368.18647231 _step‚903 ' lr-AdamW/pg1‚0.00020614994243468775 ' lr-AdamW/pg2‚0.00020614994243468775 trainer/global_step‚10999Ί^êζ#È ' lr-AdamW/pg2‚0.00020614994243468775 trainer/global_step‚10999 % _timestamp‚1.7842821760606587e+09 _runtime‚ 9368.18647231 _step‚903 ' lr-AdamW/pg1‚0.00020614994243468775‡è~ŠY2·#- _wandbruntime‚9368  _runtime‚9368ø~¶ÿ'¸#ùO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19268918 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189098815 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11614650341610 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1683417847420928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 $ _timestamp‚1.784282176061989e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87213842 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11632739304448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42952642 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1671785466494976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11632397965312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1663186581188608 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1663186564149248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1674801214490858 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1683418205799424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11763264818688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23683455 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11763176241664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380033107010048 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87213842 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚904 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87213844 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1683417847420928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1671785466494976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101874994 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380033214199296 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1671785449455616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23683721 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11614990891500 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189088836 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42952373 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1671785449455616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983ÝðÓS¹#þO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1683418205799424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11763264818688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23683455 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11763176241664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380033107010048 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87213842 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚904 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87213844 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1683417847420928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1671785466494976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101874994 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380033214199296 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1671785449455616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23683721 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11614990891500 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189088836 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42952373 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1671785449455616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19268918 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189098815 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11614650341610 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1683417847420928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 $ _timestamp‚1.784282176061989e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87213842 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11632739304448 QDDÃáMm±eviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42952642 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1671785466494976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11632397965312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1663186581188608 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1663186564149248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1674801214490858 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438ˆjŠÆ'2º#- _wandbruntime‚9368  _runtime‚9368ㆠ'»#…N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189115513 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380067559666688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189115513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1671932690485248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1683566101077504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1683566459463168 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894037 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1674948736216452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚10999 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101884046 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189105520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87221476 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842821766731942e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1683566459463168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11764200139776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87221474 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1663333091860480 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1671932707524608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11633751938560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189105520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _runtime‚9368.798986194 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1671932690485248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368303359526912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11616001945334 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23685519 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380067662654464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19270666 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23685789 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11615661395332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101884046 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42956455 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42956182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1674949093805814ÙŠÓ'¼#ŠN J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87221474 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11633410592256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11633410592256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11764284515328 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189115513 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380067559666688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189115513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1671932690485248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1683566101077504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1683566459463168 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894037 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1674948736216452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚10999 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101884046 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189105520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87221476 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842821766731942e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1683566459463168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11764200139776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87221474 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1663333091860480 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1671932707524608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11633751938560 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189105520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _runtime‚9368.798986194 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1671932690485248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368303359526912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11616001945334 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23685519 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380067662654464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19270666 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23685789 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11615661395332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101884046 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42956455 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42956182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1674949093805814 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1682177364571366 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11665266828518 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11683093467136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381755988182016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189934867 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87595442 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43143184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11665607378408 trainer/global_step‚11049 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚rù½u 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102339423 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1670512114782208 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1679147517976576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19356171 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚381755889381376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7842822174111764e+09 _step‚908 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11683434806272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23786753 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369941368694784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1690830952782848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87595444 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11814600875008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1690830594404352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1679147500937216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43142921 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1690830952782848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1682177722160616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189924888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87595444 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ìòÐó(Ë#ÿO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11683093467136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381755988182016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189934867 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87595442 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43143184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11665607378408 trainer/global_step‚11049 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102339423 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1670512114782208 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1679147517976576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19356171 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚381755889381376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7842822174111764e+09 _step‚908 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11683434806272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23786753 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369941368694784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1690830952782848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87595444 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11814600875008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1690830594404352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1679147500937216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43142921 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1690830952782848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1682177722160616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189924888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87595444 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1690830594404352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚369941387307008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87595442 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102329446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679147500937216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19356168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1682177364571366 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11665266828518 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23788733 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43146646 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189951565 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1679294741966848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11666277882240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102338498 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11684106094080 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1690978848060928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381790442033664 $ _timestamp‚1.784282218041681e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87603076 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11684447440384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87603074 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369974797453312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102348489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87603076 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87603074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19357916 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189941572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189951565 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1679294759006208 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1690979206446592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189941572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11815625968128 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11684447440384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381790332754432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19357913 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23788993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1679294741966848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169ºw¸i'Î#‰N YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11684106094080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43146909 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11666618432242 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189951565 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1679294741966848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11666277882240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102338498 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11684106094080 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1690978848060928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381790442033664 $ _timestamp‚1.784282218041681e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87603076 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11684447440384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87603074 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369974797453312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102348489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87603076 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87603074 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19357916 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153987072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189941572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189951565 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1679294759006208 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1690979206446592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189941572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11815625968128 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11684447440384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381790332754432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19357913 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23788993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1679294741966848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23788733 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43146646 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1677837648375808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚383478570612736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1677837631336448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1698243341387776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1686509569458176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190760940 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23889414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1689553514651874 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1698243341387776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11865763991552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23889680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11733788968960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚383478669413376 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1689553872241124 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11716223865316 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11734130308096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11715883315426 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102783898 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1698243699766272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371612825233408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11865844179968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1686509552418816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19443421 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚9451.696779122Ã}š(Ý#ÿO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190760940 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23889414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1689553514651874 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1698243341387776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11865763991552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23889680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11733788968960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚383478669413376 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1689553872241124 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11716223865316 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11734130308096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11715883315426 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102783898 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1698243699766272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371612825233408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11865844179968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1686509552418816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19443421 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚9451.696779122 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11734130308096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19443418 % _timestamp‚1.7842822595708013e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102783898 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87977042 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43333101 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1677837648375808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚383478570612736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1677837631336448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1698243341387776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1686509569458176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224¬I22Þ#- _wandbruntime‚9452  _runtime‚9452‰]-rVß#…N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1686656793448448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190777624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11735142942208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11717234919150 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚9452.331922529 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1698391953430016 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23891776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102802941 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1686656810487808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_”‡Ý)µpool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11716894369148 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11866788157952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43336667 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1677984142008320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190787617 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102792950 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19445163 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23891504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842822602059965e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11735142942208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1686656810487808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1677984159047680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102792950 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11866870436352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚383513023537664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43336942 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190777624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1689701393966830 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87984674 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190787617 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87984674 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102802941 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1689701036377468 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1698391953430016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87984676 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚383513124428288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87984676 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11734801595904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11734801595904 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43336667 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1677984142008320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190787617 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102792950 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19445163 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23891504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842822602059965e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11735142942208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1686656810487808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1677984159047680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102792950 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11866870436352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚383513023537664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43336942 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190777624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1689701393966830 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87984674 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190787617 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87984674 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102802941 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1689701036377468 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1698391953430016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87984676 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚383513124428288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87984676 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11734801595904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11734801595904 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269‘‡Ü2á#- _wandbruntime‚9452  _runtime‚9452*ââ#‚ 3loss/SRqqt1/assignment_loss‚0.46556210674182713 /loss/FRt1/detection_loss‚0.5062629003084179 1loss/SRqqt2/detection_loss‚0.5893742719742378 $ _timestamp‚1.784282260207702e+09 _runtime‚9452.333036893 0loss/FRt1/assignment_loss‚1.5373750010393732 0loss/FRt2/assignment_loss‚0.7703461400623428 3loss/SRqqt2/assignment_loss‚0.47555887800180424 0loss/FBt1/assignment_loss‚0.0171023567369918 1loss/FBt2/assignment_loss‚0.05130388078219941 trainer/global_step‚11099 0loss/SRqqt1/detection_loss‚0.885773943254161 0loss/FBt1/detection_loss‚0.42981374210806667 $loss/total_loss‚7.56694083920711 epoch‚27 /loss/FRt2/detection_loss‚1.2440458663315008 /loss/FBt2/detection_loss‚0.5944217518661861 _step‚9142(Z™ ã#‡ 3loss/SRqqt1/assignment_loss‚0.46556210674182713 /loss/FRt1/detection_loss‚0.5062629003084179 1loss/SRqqt2/detection_loss‚0.5893742719742378 $ _timestamp‚1.784282260207702e+09 _runtime‚9452.333036893 0loss/FRt1/assignment_loss‚1.5373750010393732 0loss/FRt2/assignment_loss‚0.7703461400623428 3loss/SRqqt2/assignment_loss‚0.47555887800180424 0loss/FBt1/assignment_loss‚0.0171023567369918 1loss/FBt2/assignment_loss‚0.05130388078219941 trainer/global_step‚11099 0loss/SRqqt1/detection_loss‚0.885773943254161 0loss/FBt1/detection_loss‚0.42981374210806667 $loss/total_loss‚7.56694083920711 epoch‚27 /loss/FRt2/detection_loss‚1.2440458663315008 /loss/FBt2/detection_loss‚0.5944217518661861 _step‚914’\wñ“ä#‚(:ˆ  ùçÒµá÷¢ gpu.0.gpu‚85.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñð¡å#‚(:÷  ùçÒÜν¦ disk.nvme0n1.in‚ 454.98046875 disk.sda.in‚0 network.sent‚ 1.7625459e+07 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 2.519265e+06 disk.drbd2165.in‚0 disk.dm-1.out‚ 369.01953125 disk.rbd3.out‚0# proc.memory.rssMB‚ 2821.83984375 proc.cpu.threads‚17 disk.sda.out‚ 369.01953125) disk./.usagePercent‚36.93500596436113' disk.nvme0c0n1.out‚40975.9423828125" disk.dm-0.out‚40975.9423828125* proc.memory.percent‚17.223143577575684 disk.rbd1.in‚0.0625! disk.rbd1.out‚107.89501953125 disk.dm-0.in‚ 454.98046875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 454.98046875 disk.dm-1.in‚0( proc.memory.availableMB‚ 439.67578125$ disk./.usageGB‚550.3582801818848 disk.sda1.out‚0 disk.sda3.out‚ 369.01953125 disk.sda3.in‚0 cpu‚26.03312709058152 disk.rbd3.in‚0% disk.nvme0n1.out‚40975.9423828125$ memory_percent‚97.31643199920654H—Ç’æ#‚(:‡ ¯ùçÒ¨Ÿ¹¢ gpu.0.gpu‚28.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÃ©óç#‚(:è ¯ùçÒÆìê¦ disk.dm-0.in‚ 455.6953125 disk.rbd0.in‚0$ memory_percent‚97.31760025024414# proc.memory.rssMB‚ 2821.83984375! disk.nvme0c0n1.in‚ 455.6953125 disk.sda1.in‚0& proc.memory.availableMB‚ 439.484375 proc.cpu.threads‚17! disk.rbd1.out‚107.95751953125 disk.dm-0.out‚ 42090.3515625 disk.sda3.out‚ 369.23828125( disk./.usagePercent‚36.9349975161125$ disk.nvme0c0n1.out‚ 42090.3515625 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625# disk./.usageGB‚550.358154296875 disk.nvme0n1.in‚ 455.6953125 network.recv‚ 2.524628e+06 disk.sda2.out‚0 disk.rbd3.in‚0" disk.nvme0n1.out‚ 42090.3515625 disk.rbd0.out‚0 cpu‚26.032633304526364 network.sent‚ 1.7675513e+07 disk.dm-1.out‚ 369.23828125 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda.out‚ 369.23828125 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.223143577575684 disk.dm-1.in‚0£¿O2è#- _wandbruntime‚9494  _runtime‚9494¸'õ€Éé#à ' lr-AdamW/pg2‚0.00020052360898349653 trainer/global_step‚11149 % _timestamp‚1.7842823020093806e+09 _runtime‚ 9494.13508962 _step‚915 ' lr-AdamW/pg1‚0.000200523608983496536I¾MÎê#È _step‚915 ' lr-AdamW/pg1‚0.00020052360898349653 ' lr-AdamW/pg2‚0.00020052360898349653 trainer/global_step‚11149 % _timestamp‚1.7842823020093806e+09 _runtime‚ 9494.13508962“×ÈÖN2ë#- _wandbruntime‚9494  _runtime‚9494È„ª}%ì#ûO KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19530671 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1705656446749696 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103248327 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1705656446749696 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1705656088371200 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1696929664732382 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88358644 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842823020106945e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1693871620939776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43523334 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191606971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1696930022321632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1685163181969408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11784825809920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19530668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11917075862528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373284263159808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1705656088371200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11784484470784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43523055 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11766499802334 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11784825809920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385201229735936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11766840352224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103238350 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88358644 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373284244547584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11784484470784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191596992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11916985188352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1693871620939776 _step‚916 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23992387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23992387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1705656088371200 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1696929664732382 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88358644 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842823020106945e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1693871620939776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43523334 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191606971 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1696930022321632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1685163181969408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11784825809920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19530668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11917075862528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373284263159808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1705656088371200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11784484470784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43523055 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11766499802334 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11784825809920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385201229735936 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11766840352224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103238350 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88358644 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373284244547584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11784484470784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191596992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11916985188352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1693871620939776 _step‚916”KQ}‘2î#- _wandbruntime‚9494  _runtime‚9494Fp.- 'ï#…N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11918092159488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373317691918336 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1697077544047338 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191613676 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1697077186457976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43527076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1694018844930048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11785497097728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191623669 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103257393 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11785838444032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1705804700413440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11785838444032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 % _timestamp‚1.7842823026451867e+09 _step‚917 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1694018861969408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43526805 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚385235668507136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191613676 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191623669 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103247402 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1694018861969408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11785497097728 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1705804342027776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19532413 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385235784077824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11917995201024 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1694018844930048 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1685309692641280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88366276 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1705804700413440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 trainer/global_step‚11149 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23994660 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191613676 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1697077186457976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43527076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1694018844930048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11785497097728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191623669 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103257393 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11785838444032 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1705804700413440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11785838444032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 % _timestamp‚1.7842823026451867e+09 _step‚917 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1694018861969408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43526805 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚385235668507136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191613676 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191623669 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103247402 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1694018861969408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11785497097728 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1705804342027776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19532413 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385235784077824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11917995201024 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1694018844930048 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1685309692641280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88366276 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1705804700413440 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 trainer/global_step‚11149 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23994660 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386923699160064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1713069193733120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11968016686080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11817116289242 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11817456839132 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713068835354624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192433044 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11835179972608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43712353 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88740242 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11835521311744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713069193733120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88740244 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103692802 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103702779 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1701233672421376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11835521311744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11835179972608 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1692488698523648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103702779 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43712086 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1701233672421376¹”b(ÿ#€P WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19617918 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1704305814812890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1704306172402140 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153987072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713068835354624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192433044 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11835179972608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43712353 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88740242 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11835521311744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713069193733120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88740244 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103692802 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103702779 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1701233672421376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11835521311744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11835179972608 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1692488698523648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103702779 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43712086 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1701233672421376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚11199 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713068835354624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24094168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚9536.558676303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1701233655382016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192443023 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19617921 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11968098971648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386923699160064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1713069193733120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11968016686080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11817116289242 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11817456839132 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386958250905088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374989111232512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1701380913451008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103711845 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103701854 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103711845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43715868 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192449728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1713217447396864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24096473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11836533945856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192459721 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11969121060352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1713217447396864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1701380913451008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24096205 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1692635226234880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1713217089011200 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88747876 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103701854 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1704453694127846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11836192599552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 $ _timestamp‚1.784282345060525e+09 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1692635209195520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43716139 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚9537.186571455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19619663 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192459721 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192449728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0nƒ6'‚$‰N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386958135334400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88747876 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103701854 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1704453694127846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11836192599552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 $ _timestamp‚1.784282345060525e+09 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1692635209195520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43716139 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚9537.186571455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19619663 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192459721 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192449728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386958250905088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374989111232512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1701380913451008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103711845 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103701854 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103711845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43715868 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192449728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1713217447396864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24096473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11836533945856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192459721 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11969121060352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1713217447396864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1701380913451008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24096205 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1692635226234880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1713217089011200 E&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16716328263282776 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23581136763095856 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6638278365135193 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9911280870437622 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24709002673625946 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7018946409225464 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6243958473205566 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5139542818069458 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23068900406360626 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30329135060310364 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5916450023651123 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.757520854473114 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6825488209724426 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045282091945409775 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.907334566116333 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3039546608924866 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6992794871330261 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6498433351516724 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5017242431640625 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.721123993396759 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2934482991695404 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.66090327501297 *jet/accuracy_2_of_3‚0.5764600038528442 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969645738601685 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7452927827835083 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8634832501411438 _step‚923 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.681625485420227 5 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27432781457901 jet/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5399084687232971 ,jet/accuracy_4_of_4‚0.006769374944269657 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993476867675781 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4642857015132904 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7218064069747925 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22250865399837494 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7215469479560852 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 6validation_average_jet_accuracy‚0.3924371600151062 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5549999475479126 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.8093154430389404 particle/accuracy_2_of_2‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.46875 particle/accuracy_4_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5245264172554016 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7888420820236206 +jet/accuracy_3_of_3‚0.07030312716960907 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 epoch‚27 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.038889966905117035 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7408276796340942 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0389605388045311 particle/accuracy_5_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18356187641620636 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6861661076545715 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9453757405281067 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2588237524032593 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6666479110717773 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7707861661911011 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.651871919631958 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.73762047290802 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2921183705329895 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2366916537284851 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9967206120491028 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8518518209457397 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7174011468887329 *jet/accuracy_1_of_4‚0.9729810357093811 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7695534229278564 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.667048990726471 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989920258522034 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6536337733268738 +jet/accuracy_3_of_4‚0.07658954709768295 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28832554817199707 % _timestamp‚1.7842823632729256e+09 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02134215645492077 +jet/accuracy_2_of_2‚0.16060519218444824 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3478051722049713 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6447750926017761 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7832920551300049 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28219541907310486 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022725870832800865 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7290013432502747 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17169520258903503 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07310938835144043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.8036505579948425 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7934618592262268 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891728162765503 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 jet/accuracy_1_of_5‚1 7"Purity/*FRt*SRqqt2FBt/event_purity‚0.87164705991745 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 trainer/global_step‚11199 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_1‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25484269857406616 'particle/accuracy‚0.844590961933136 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976863861083984 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8521735668182373 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3866667151451111 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5202818512916565 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7167935371398926 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6788071393966675 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7934618592262268 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0224966611713171 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25008153915405273 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 /particle/accuracy_3_of_4‚0.9979296326637268 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3239844739437103 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2934482991695404 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6029536128044128 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7849584221839905 'particle/f_score‚0.6310341954231262 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15916667878627777 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28404390811920166 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993476867675781 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947041273117065 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.155614972114563 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35729390382766724 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8518518209457397 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2402673214673996 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6414902210235596 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6536337733268738 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5103524327278137 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7630462646484375 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6823150515556335 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021560663357377052 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9295526146888733 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +particle/sensitivity‚0.6189211010932922 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7360394597053528 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_1‚0.3539142608642578 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.022899169474840164 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974802732467651 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7887616157531738 particle/accuracy_2_of_5‚1 _runtime‚ 9555.3995749 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0224966611713171 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.659549C^€ŸL5939254761 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719ê•«tp=Œ$êz :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28404390811920166 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993476867675781 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947041273117065 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.155614972114563 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35729390382766724 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8518518209457397 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2402673214673996 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6414902210235596 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6536337733268738 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5103524327278137 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7630462646484375 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6823150515556335 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021560663357377052 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9295526146888733 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +particle/sensitivity‚0.6189211010932922 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7360394597053528 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_1‚0.3539142608642578 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.022899169474840164 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974802732467651 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7887616157531738 particle/accuracy_2_of_5‚1 _runtime‚ 9555.3995749 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0224966611713171 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6595495939254761 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2615407109260559 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04951805621385574 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0855606272816658 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22123277187347412 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3043614327907562 particle/accuracy_3_of_3‚1 )loss/val_total_loss‚7.903291247751019 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992644786834717 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7205444574356079 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8634832501411438 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944548606872559 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7215793132781982 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49768516421318054 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9063029289245605 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968388080596924 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9330615997314453 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3872687816619873 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7886282801628113 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41425222158432007 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23808488249778748 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16716328263282776 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23581136763095856 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6638278365135193 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9911280870437622 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24709002673625946 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7018946409225464 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6243958473205566 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5139542818069458 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23068900406360626 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30329135060310364 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5916450023651123 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.757520854473114 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6825488209724426 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045282091945409775 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.907334566116333 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3039546608924866 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6992794871330261 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6498433351516724 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5017242431640625 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.721123993396759 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2934482991695404 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.66090327501297 *jet/accuracy_2_of_3‚0.5764600038528442 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969645738601685 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7452927827835083 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8634832501411438 _step‚923 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.681625485420227 5 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27432781457901 jet/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5399084687232971 ,jet/accuracy_4_of_4‚0.006769374944269657 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993476867675781 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4642857015132904 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7218064069747925 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22250865399837494 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7215469479560852 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 6validation_average_jet_accuracy‚0.3924371600151062 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5549999475479126 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.8093154430389404 particle/accuracy_2_of_2‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.46875 particle/accuracy_4_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5245264172554016 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7888420820236206 +jet/accuracy_3_of_3‚0.07030312716960907 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 epoch‚27 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.038889966905117035 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7408276796340942 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0389605388045311 particle/accuracy_5_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18356187641620636 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6861661076545715 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9453757405281067 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2588237524032593 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6666479110717773 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7707861661911011 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.651871919631958 7"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.73762047290802 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2921183705329895 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2366916537284851 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9967206120491028 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8518518209457397 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7174011468887329 *jet/accuracy_1_of_4‚0.9729810357093811 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7695534229278564 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.667048990726471 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989920258522034 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6536337733268738 +jet/accuracy_3_of_4‚0.07658954709768295 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28832554817199707 % _timestamp‚1.7842823632729256e+09 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02134215645492077 +jet/accuracy_2_of_2‚0.16060519218444824 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3478051722049713 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6447750926017761 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7832920551300049 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28219541907310486 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022725870832800865 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7290013432502747 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17169520258903503 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07310938835144043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.8036505579948425 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7934618592262268 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891728162765503 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 jet/accuracy_1_of_5‚1 7"Purity/*FRt*SRqqt2FBt/event_purity‚0.87164705991745 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 trainer/global_step‚11199 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_1‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25484269857406616 'particle/accuracy‚0.844590961933136 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976863861083984 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8521735668182373 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3866667151451111 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5202818512916565 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7167935371398926 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6788071393966675 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7934618592262268 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0224966611713171 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25008153915405273 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 /particle/accuracy_3_of_4‚0.9979296326637268 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3239844739437103 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2934482991695404 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6029536128044128 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7849584221839905 'particle/f_score‚0.6310341954231262 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15916667878627777 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129›°Õ[ã‘$‚(:† ‰úçÒœÄû¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ÂÒýŽ$‚(:ò ‰úçÒ»ºÐ¥ disk.sda.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 439.21484375 disk.sda2.out‚0# disk.dm-0.out‚46332.66162109375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2821.83984375 disk.sda.out‚ 370.6953125 disk.nvme0n1.in‚ 459.609375& disk.nvme0n1.out‚46332.66162109375 cpu‚26.036904461751966 proc.cpu.threads‚17 network.recv‚ 2.548333e+06$ disk./.usageGB‚550.7525405883789 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 459.609375 disk.sda1.in‚0( disk.nvme0c0n1.out‚46332.66162109375* proc.memory.percent‚17.223143577575684 network.sent‚ 1.7821905e+07) disk./.usagePercent‚36.96146511104747 disk.dm-1.in‚0 disk.dm-1.out‚ 370.6953125 disk.rbd3.out‚0 disk.rbd0.out‚0$ memory_percent‚97.31924533843994! disk.rbd1.out‚108.20751953125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 459.609375 disk.sda3.out‚ 370.6953125 disk.rbd1.in‚0.0625 disk.sda1.out‚0Ú”“$‚(:ˆ ˜úçÒòø¢ gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚270.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾bE|ô$‚(:é ˜úçÒÜ–Ø¥ disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚550.7522163391113! disk.nvme0n1.out‚ 47386.046875 disk.sda.in‚0 disk.sda3.out‚ 370.87890625 disk.rbd3.out‚0* proc.memory.percent‚17.223143577575684 cpu‚26.03640497666055 disk.sda.out‚ 370.87890625 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 460.32421875" disk.nvme0c0n1.in‚ 460.32421875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.out‚ 47386.046875 disk.rbd0.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2821.83984375# disk.nvme0c0n1.out‚ 47386.046875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 370.87890625 disk.rbd3.in‚0' proc.memory.availableMB‚ 439.4296875 network.recv‚ 2.55321e+06) disk./.usagePercent‚36.96144335040706 disk.dm-0.in‚ 460.32421875 network.sent‚ 1.7881453e+07 disk.sda1.in‚0 disk.sda2.in‚0! disk.rbd1.out‚108.20751953125$ memory_percent‚97.31793403625488{Õ2‘$- _wandbruntime‚9596  _runtime‚9596…'¤Ê’$Ä trainer/global_step‚11249 % _timestamp‚1.7842824048211102e+09 _runtime‚9596.946727224 _step‚924 ' lr-AdamW/pg1‚0.00019678401945035802 ' lr-AdamW/pg2‚0.00019678401945035802–¤ú*Ï“$É trainer/global_step‚11249 % _timestamp‚1.7842824048211102e+09 _runtime‚9596.946727224 _step‚924 ' lr-AdamW/pg1‚0.00019678401945035802 ' lr-AdamW/pg2‚0.00019678401945035802œ€â¹ 2”$- _wandbruntime‚9596  _runtime‚9596j=M(•$úO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11903950019072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89242510 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1722429240572416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89242508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388680530475520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193460084 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11885452501379 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1710525273513984 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104217576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1701722895331328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1710525290553344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1722428882193920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193470063 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12037401271808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚388680433772032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842824048221827e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376643051112448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1713608705421957 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24228605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1710525290553344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11903608679936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚11249 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1710525273513984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1701722912370688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89242508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19707688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24228347 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12037479363072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚9596.947939853 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376643032500224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19707691 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11885793051269##°Þ –$ÿO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1710525290553344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11903608679936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚11249 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1710525273513984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1701722912370688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89242508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19707688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24228347 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12037479363072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚9596.947939853 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376643032500224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19707691 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11885793051269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11903950019072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89242510 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1722429240572416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89242508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388680530475520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193460084 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11885452501379 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1710525273513984 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104217576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1701722895331328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1710525290553344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1722428882193920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193470063 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12037401271808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚388680433772032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842824048221827e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376643051112448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1713608705421957 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24228605 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976$ž@R2—$- _wandbruntime‚9597  _runtime‚9597½qL '˜$…N Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12038508258816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1710672531582976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104226628 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1710672531582976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388714988129792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388714883044864 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104236619 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1713756227147663 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 % _timestamp‚1.7842824054320889e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89250142 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89250140 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43939833 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104236619 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1710672514543616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11904621306880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12038421786112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1722577494236160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24230675 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚9597.557860339 _step‚926 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11886804105103 trainer/global_step‚11249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193476768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24230400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11904621306880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19709433 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193486761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 238425210882¿Á'™$ŠN YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12038508258816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1710672531582976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104226628 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1710672531582976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388714988129792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388714883044864 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104236619 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1713756227147663 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 % _timestamp‚1.7842824054320889e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89250142 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89250140 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43939833 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104236619 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1710672514543616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11904621306880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12038421786112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1722577494236160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24230675 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚9597.557860339 _step‚926 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11886804105103 trainer/global_step‚11249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193476768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24230400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11904621306880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19709433 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193486761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3ž¦j ,2š$- _wandbruntime‚9597  _runtime‚9597ž}›$‰ 3loss/FBt1/assignment_loss‚0.0002880489895406729 /loss/FRt2/detection_loss‚1.2222037669550545 1loss/SRqqt1/detection_loss‚1.0243130515208425 trainer/global_step‚11249 _runtime‚9597.561403708 3loss/SRqqt2/assignment_loss‚0.43234240299678517 0loss/FRt1/detection_loss‚0.48866722170942395 0loss/FBt2/detection_loss‚0.39669982510039936 &loss/total_loss‚7.6068990793400335 /loss/FRt1/assignment_loss‚1.523137852928396 1loss/FBt2/assignment_loss‚0.04039106866559694 epoch‚28 _step‚927 0loss/FRt2/assignment_loss‚0.9575172947709907 1loss/SRqqt2/detection_loss‚0.6706371672914032 0loss/FBt1/detection_loss‚0.48768996281874866 % _timestamp‚1.7842824054360483e+09 2loss/SRqqt1/assignment_loss‚0.3630114155928521F õ;œ$Ž /loss/FRt1/assignment_loss‚1.523137852928396 1loss/FBt2/assignment_loss‚0.04039106866559694 epoch‚28 _step‚927 0loss/FRt2/assignment_loss‚0.9575172947709907 1loss/SRqqt2/detection_loss‚0.6706371672914032 0loss/FBt1/detection_loss‚0.48768996281874866 % _timestamp‚1.7842824054360483e+09 2loss/SRqqt1/assignment_loss‚0.3630114155928521 3loss/FBt1/assignment_loss‚0.0002880489895406729 /loss/FRt2/detection_loss‚1.2222037669550545 1loss/SRqqt1/detection_loss‚1.0243130515208425 trainer/global_step‚11249 _runtime‚9597.561403708 3loss/SRqqt2/assignment_loss‚0.43234240299678517 0loss/FRt1/detection_loss‚0.48866722170942395 0loss/FBt2/detection_loss‚0.39669982510039936 &loss/total_loss‚7.6068990793400335Ÿpm~C$‚(:’ §úçÒžÊÓ¢ gpu.0.gpu‚90.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T7‡ÿž$‚(:ô §úçÒ¬‚ù¥! disk.rbd1.out‚108.27001953125$ memory_percent‚97.31941223144531 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0* proc.memory.percent‚17.223143577575684 network.recv‚ 2.556803e+06 disk.dm-0.in‚ 461.15234375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 373.70703125 disk.dm-1.in‚0 disk.dm-1.out‚ 373.70703125 network.sent‚ 1.7897458e+07$ disk./.usageGB‚548.7767448425293 disk.rbd3.out‚0 disk.nvme0n1.in‚ 461.15234375# proc.memory.rssMB‚ 2821.83984375" disk.nvme0c0n1.in‚ 461.15234375 disk.sda2.out‚0$ proc.memory.availableMB‚439.1875 proc.cpu.threads‚17 disk.sda1.out‚0' disk.nvme0c0n1.out‚47397.3544921875 disk.sda3.in‚0 cpu‚26.036061773490363) disk./.usagePercent‚36.82886780074045 disk.rbd3.in‚0" disk.dm-0.out‚47397.3544921875% disk.nvme0n1.out‚47397.3544921875 disk.sda.out‚ 373.70703125 disk.sda2.in‚0 disk.rbd1.in‚0.06250«TŠ“Ÿ$‚(:ˆ ¶úçÒ¬„Ë¢ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z”+¾ý $‚(:ò ¶úçÒ‰Ðü¤ disk.drbd2165.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚47408.3212890625% disk.nvme0n1.out‚47408.3212890625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 461.15234375 disk.sda.out‚ 375.203125 disk.dm-0.in‚ 461.15234375" disk.dm-0.out‚47408.3212890625 disk.sda1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2821.83984375* proc.memory.percent‚17.223143577575684$ disk./.usageGB‚548.7741775512695 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 375.203125 disk.rbd3.in‚0! disk.rbd1.out‚108.27001953125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 461.15234375( proc.memory.availableMB‚ 438.87109375 cpu‚26.035648894882605 disk.sda3.out‚ 375.203125 disk.sda.in‚0 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚97.32134342193604 network.sent‚ 1.7947591e+07 network.recv‚ 2.56185e+06* disk./.usagePercent‚36.828695507669906ÏsÕW“¡$‚(:ˆ ÅúçÒüﺤ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ª½­¢$‚(:ó ÅúçÒÕ–Ö¥ disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 439.10546875 network.recv‚ 2.56563e+06! disk.nvme0c0n1.in‚ 461.8671875 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 375.73046875 disk.dm-1.out‚ 375.73046875 disk.rbd3.out‚0) disk./.usagePercent‚36.82873493283017 disk.sda.out‚ 375.73046875 network.sent‚ 1.795264e+07 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚47416.0458984375$ memory_percent‚97.31991291046143 cpu‚26.035202211484933 disk.sda1.in‚0' disk.nvme0c0n1.out‚47416.0458984375 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 461.8671875" disk.dm-0.out‚47416.0458984375* proc.memory.percent‚17.22j$¤`á3143577575684$ disk./.usageGB‚548.7747650146484 disk.drbd2165.in‚0! disk.rbd1.out‚108.33251953125 disk.nvme0n1.in‚ 461.8671875# proc.memory.rssMB‚ 2821.83984375 disk.dm-1.in‚0 proc.cpu.threads‚17%²…b2£$- _wandbruntime‚9636  _runtime‚9636)˘Hʤ$Ä _runtime‚9636.705860154 _step‚928 ' lr-AdamW/pg1‚0.00019491797501826165 ' lr-AdamW/pg2‚0.00019491797501826165 trainer/global_step‚11299 % _timestamp‚1.7842824445802696e+09‡ˆ¦eÏ¥$É _runtime‚9636.705860154 _step‚928 ' lr-AdamW/pg1‚0.00019491797501826165 ' lr-AdamW/pg2‚0.00019491797501826165 trainer/global_step‚11299 % _timestamp‚1.7842824445802696e+09 ¹+.½2¦$- _wandbruntime‚9636  _runtime‚9636è(§$ûO ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1717887324995584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚11299 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1729841629177344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1717887342034944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12088668714496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1717887342034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1729841629177344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89624108 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1729841987555840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89624110 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12088761485824 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1709048428924928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44127149 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1717887324995584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚9636.707395803 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89624108 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1729841987555840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11954304181760 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1720984855502465 % _timestamp‚1.7842824445815358e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390403139141120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44126878 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11954645520896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104682005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19794941 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚929 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194306115 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1720984497913215 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11954304181760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104682005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11954645520896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24331940 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11936068988287 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24332208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89624110 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378314489038848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1709048445964288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169½Ì‡Æ(¨$€P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12088668714496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1717887342034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1729841629177344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89624108 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1729841987555840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89624110 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12088761485824 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1709048428924928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44127149 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1717887324995584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚9636.707395803 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89624108 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1729841987555840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11954304181760 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1720984855502465 % _timestamp‚1.7842824445815358e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390403139141120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44126878 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11954645520896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104682005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19794941 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚929 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194306115 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1720984497913215 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11954304181760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104682005 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11954645520896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24331940 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11936068988287 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24332208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89624110 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378314489038848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1709048445964288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1717887324995584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚11299 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1729841629177344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1717887342034944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888¡ŒŒÃ2©$- _wandbruntime‚9637  _runtime‚9637’J- 'ª$„N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24334051 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1709194939596800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104681080 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104691071 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1718034566025216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194312820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390437588789248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44130734 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1709194956636160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390437698068480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚9637.318607549 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1729990241219584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1729990241219584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194322813 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89631742 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1718034566025216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1721132377228171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11955316808704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19796686 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1721132019638809 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12089780271104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 $ _timestamp‚1.784282445192828e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194322813 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11955658155008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11955316808704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1718034583064576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378347899185152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89631740 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12089689604096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚930 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194312820 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1729989882833920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89631742 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104691071 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378347917797376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11937080042009 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19796683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1718034583064576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0øÈÅ­™«$‰N YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11955316808704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19796686 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1721132019638809 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12089780271104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 $ _timestamp‚1.784282445192828e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194322813 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11955658155008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11955316808704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceœö=dv!StatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1718034583064576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378347899185152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89631740 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12089689604096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚930 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194312820 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1729989882833920 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89631742 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104691071 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378347917797376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11937080042009 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19796683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1718034583064576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24334051 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1709194939596800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104681080 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104691071 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1718034566025216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194312820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390437588789248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44130734 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1709194956636160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390437698068480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚9637.318607549 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1729990241219584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1729990241219584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194322813 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89631742 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1718034566025216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1721132377228171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¢hË2¬$- _wandbruntime‚9637  _runtime‚9637Œ™šž ­$‡ 3loss/FBt2/assignment_loss‚0.0017049880351580687 /loss/FBt1/detection_loss‚0.4809859898301261 epoch‚28 % _timestamp‚1.7842824451962848e+09 0loss/FRt1/assignment_loss‚1.4418772073161654 0loss/FRt2/assignment_loss‚0.7696782525847939 3loss/SRqqt1/assignment_loss‚0.36950986774044725 /loss/FRt1/detection_loss‚0.5090736559908012 %loss/total_loss‚7.535671278147584 _runtime‚9637.321615698 2loss/SRqqt2/assignment_loss‚0.4564995795815708 1loss/SRqqt2/detection_loss‚0.6556264479409318 trainer/global_step‚11299 2loss/FBt1/assignment_loss‚0.010922888814464855 /loss/FRt2/detection_loss‚1.2684857222238843 1loss/SRqqt1/detection_loss‚0.7976681024509858 /loss/FBt2/detection_loss‚0.7736385756382539 _step‚931nnÚ®$Œ 2loss/FBt1/assignment_loss‚0.010922888814464855 /loss/FRt2/detection_loss‚1.2684857222238843 1loss/SRqqt1/detection_loss‚0.7976681024509858 /loss/FBt2/detection_loss‚0.7736385756382539 _step‚931 3loss/FBt2/assignment_loss‚0.0017049880351580687 /loss/FBt1/detection_loss‚0.4809859898301261 epoch‚28 % _timestamp‚1.7842824451962848e+09 0loss/FRt1/assignment_loss‚1.4418772073161654 0loss/FRt2/assignment_loss‚0.7696782525847939 3loss/SRqqt1/assignment_loss‚0.36950986774044725 /loss/FRt1/detection_loss‚0.5090736559908012 %loss/total_loss‚7.535671278147584 _runtime‚9637.321615698 2loss/SRqqt2/assignment_loss‚0.4564995795815708 1loss/SRqqt2/detection_loss‚0.6556264479409318 trainer/global_step‚11299£ p'È“¯$‚(:ˆ ÔúçÒðণ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'jXîù°$‚(:î ÔúçÒ–·È¤ disk.sda.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 438.87109375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.7957641e+07) disk./.usagePercent‚36.82849761384593 disk.rbd1.in‚0.0625# disk.nvme0n1.out‚47428.19140625* proc.memory.percent‚17.223143577575684% disk.nvme0c0n1.out‚47428.19140625 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 network.recv‚ 2.568789e+06! disk.nvme0c0n1.in‚ 462.6953125 disk.dm-1.out‚ 376.08984375 proc.cpu.threads‚17$ disk./.usageGB‚548.7712287902832 disk.sda2.out‚0 disk.sda3.out‚ 376.08984375 disk.rbd3.in‚0 disk.dm-0.in‚ 462.6953125 cpu‚26.03471530045767 disk.rbd0.in‚0 disk.sda.out‚ 376.08984375 disk.nvme0n1.in‚ 462.6953125$ memory_percent‚97.32134342193604# proc.memory.rssMB‚ 2821.83984375 disk.dm-1.in‚0! disk.rbd1.out‚108.33251953125 disk.dm-0.out‚47428.191406253ïk»’±$‚(:‡ ãúçÒè£ê¢ gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=É,™ë²$‚(:à ãúçÒ€÷¸¤ disk.nvme0c0n1.in‚ 463.40625 disk.sda3.in‚0 disk.nvme0n1.in‚ 463.40625& proc.memory.availableMB‚ 438.921875 proc.cpu.threads‚17% disk.nvme0n1.out‚47435.3115234375 disk.sda1.out‚0 network.recv‚ 2.574444e+06 disk.sda.out‚376.9375 disk.sda3.out‚376.9375 disk.dm-1.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚108.39501953125 disk.dm-0.in‚ 463.40625 network.sent‚ 1.8020856e+07 disk.sda.in‚0 disk.sda2.in‚0" disk.dm-0.out‚47435.3115234375 disk.sda1.in‚0 disk.drbd2165.in‚0 cpu‚26.034275937595368' disk.nvme0c0n1.out‚47435.3115234375 disk.sda2.out‚0* proc.memory.percent‚17.223143577575684$ disk./.usageGB‚548.7714958190918 disk.rbd3.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0# memory_percent‚97.3210334777832# proc.memory.rssMB‚ 2821.83984375 disk.rbd0.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚36.82851553437334 disk.dm-1.out‚376.9375½ì‡²‘³$‚(:† òúçÒâ©ã£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$Dåªï´$‚(:ä òúçÒ˜…ˆ¨ disk.rbd0.in‚0 network.sent‚ 1.8025852e+07) disk./.usagePercent‚36.82810003414526# disk.nvme0n1.out‚47446.08203125$ memory_percent‚97.32263088226318 disk.nvme0c0n1.in‚ 464.234375 disk.sda3.in‚0 disk.dm-1.out‚ 377.109375! disk.rbd1.out‚108.39501953125 disk.nvme0n1.in‚ 464.234375 disk.sda1.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚47446.08203125 disk.sda2.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 438.66015625 disk.dm-0.out‚47446.08203125 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 cpu‚26.0337912565988 network.recv‚ 2.577776e+06$ disk./.usageGB‚548.7653045654297 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 377.109375 disk.dm-0.in‚ 464.234375 disk.sda.out‚ 377.109375 disk.sda2.in‚0 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2821.83984375* proc.memory.percent‚17.223143577575684 proc.cpu.threads‚17A2µ$- _wandbruntime‚9676  _runtime‚9676cZsʶ$Ä ' lr-AdamW/pg1‚0.00019305459066692776 ' lr-AdamW/pg2‚0.00019305459066692776 trainer/global_step‚11349 % _timestamp‚1.7842824845049949e+09 _runtime‚9676.630562796 _step‚932:Û)Ï·$É _step‚932 ' lr-AdamW/pg1‚0.00019305459066692776 ' lr-AdamW/pg2‚0.00019305459066692776 trainer/global_step‚11349 % _timestamp‚1.7842824845049949e+09 _runtime‚9676.630562796¤µÑg2¸$- _wandbruntime‚9676  _runtime‚9676ûLà(¹$ûO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90005708 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1737254376160768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19882188 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1737254734539264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105136457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1716373979557888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11986685475195 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚11349 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105126480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12005341022720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392125929122816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90005710 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12004999683584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392125824030720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105126480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44317723 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19882191 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1728360647993723 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1725249393516544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195132188 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1725249393516544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚9676.631762391 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚933 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1737254376160768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90005708 % _timestamp‚1.7842824845059962e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12004999683584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379985926965248 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1737254734539264 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1716373962518528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12139915677696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1725249376477184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105136457 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195142167 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24435535 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1737254734539264 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1716373962518528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12139915677696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1725249376477184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105136457 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195142167 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24435535 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1737254376160768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19882188 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1737254734539264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105136457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1716373979557888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 OIDek”%áÄ viceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11986685475195 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚11349 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105126480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12005341022720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392125929122816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90005710 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12004999683584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392125824030720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105126480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44317723 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¥)É(?2»$- _wandbruntime‚9677  _runtime‚9677ütƒ '¼$„N E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44321849 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1725396617506816 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19883936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90013342 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90013342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚392160380617216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195148872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1728508169719317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842824851166613e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105135532 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚11349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12006353656832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12006353656832 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24437658 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1716520490229760 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195158865 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105145523 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44321591 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19883933 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1737402988203008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90013340 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392160281823744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1728508527308679 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11987696528917 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1716520473190400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12140944712192 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195158865 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105145523 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44321591 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19883933 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1737402988203008 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90013340 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392160281823744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1728508527308679 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11987696528917 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1716520473190400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12140944712192 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19883936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90013342 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90013342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚392160380617216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195148872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1728508169719317 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842824851166613e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105135532 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚11349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12006353656832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12006353656832 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24437658 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1716520490229760 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12037301962103 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44508298 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1744667481522688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 % _timestamp‚1.7842825242479532e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1723699496112128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393848264053248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44508034 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195978219 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11399 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24538857 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1735736798074231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90387308 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12191004253696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381657346279424 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1735737155663481 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1732611427958784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90387308 `MDeviceSbw[åßtatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12190917773824 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1732611444998144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12056036524544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1732611444998144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90387310 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24538596 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1744667123144192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1744667123144192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12055695185408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105580932 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12037642511993 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393848369145344Ðîzá(Ê$€P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44508298 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1744667481522688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 % _timestamp‚1.7842825242479532e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1723699496112128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393848264053248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44508034 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195978219 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11399 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24538857 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1735736798074231 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90387308 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12191004253696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381657346279424 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1735737155663481 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1732611427958784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90387308 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12190917773824 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1732611444998144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12056036524544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1732611444998144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90387310 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24538596 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1744667123144192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1744667123144192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12055695185408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105580932 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12037642511993 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393848369145344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105580932 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19969441 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12037301962103©¿wæ2Ë$- _wandbruntime‚9716  _runtime‚9716S©&ü 'Ì$…N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1732758668988416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12038313015825 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1744815735186432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚381690775037952 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1732758686027776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12056707812352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1723846006784000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393882703877120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105589984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393882817350656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12192023700480 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1735884677389187 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195984924 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚938 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105599975 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44512090 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195994917 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1744815376800768 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44511813 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195994917 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90394942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1723846023823360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90394940 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90394940 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24540630 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12191928839168 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1732758668988416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7842825248592114e+09õðÕ»Í$ŠN _runtime‚9716.985021935 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105589984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105599975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19971186 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1732758686027776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90394942 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12057049158656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1744815735186432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1735884319799825 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12038653565827 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24540904 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12056707812352 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44512090 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195994917 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1744815376800768 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44511813 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195994917 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90394942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1723846023823360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90394940 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90394940 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24540630 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12191928839168 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1732758668988416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7842825248592114e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1732758668988416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12038313015825 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1744815735186432 REDeviceStatsMonitor.on_N—J­U train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚381690775037952 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1732758686027776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12056707812352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1723846006784000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393882703877120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105589984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚393882817350656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12192023700480 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1735884677389187 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195984924 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚938 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105599975 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚383328784205824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7842825642120042e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1752079870127616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12106390687232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24641726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24641441 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12087918449011 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚383328802818048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1743113305743989 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11449 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90768908 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1731025029705728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106045361 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1739973479440384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚941 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44698129 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12241966954496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1739973496479744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1752080228506112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12242059725824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1739973496479744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12106390687232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196814271 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90768908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106035384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12106732026368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1752079870127616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12106390687232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24641726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24641441 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12087918449011 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚383328802818048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1743113305743989 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11449 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90768908 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1731025029705728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106045361 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1739973479440384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚941 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44698129 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12241966954496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1739973496479744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1752080228506112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12242059725824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1739973496479744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12106390687232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196814271 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90768908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106035384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12106732026368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚383328784205824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0­•„2Ý$- _wandbruntime‚9756  _runtime‚9756Á5‡Ý Þ$…N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1743260469880333 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90776542 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1752228482169856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚383362231576576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12107744660480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106054427 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1731171557416960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1740120737509376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12107403314176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceSÄø¼Ú.tatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395605313036800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1740120737509376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚383362212964352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90776540 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196820976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12243081460224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7842825648230944e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90776540 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1740120720470016 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24643549 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395605197466112 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1752228482169856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44702243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚11449 _step‚942 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196830969 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24643807 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12242984501760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196830969 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1740120720470016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106044436 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44701982 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106044436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20058433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12107744660480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90776542 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1743260827469695 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12089270052735 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚04wçé'ß$ŠN OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106044436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20058433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12107744660480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90776542 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1743260827469695 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12089270052735 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1743260469880333 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90776542 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1752228482169856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚383362231576576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12107744660480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106054427 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1731171557416960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1740120737509376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12107403314176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395605313036800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1740120737509376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚383362212964352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90776540 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196820976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12243081460224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7842825648230944e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90776540 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1740120720470016 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24643549 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395605197466112 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1752228482169856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44702243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚11449 _step‚942 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196830969 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24643807 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12242984501760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196830969 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1740120720470016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106044436 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44701982 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968®Ñý‘2à$- _wandbruntime‚9756  _runtime‚9756®E’ á$† _step‚943 3loss/SRqqt1/assignment_loss‚0.42400955909383153 3loss/SRqqt2/assignment_loss‚0.49249891165299026 epoch‚28 _runtime‚ 9756.95149455 %loss/total_loss‚7.920983556502408 /loss/FRt1/detection_loss‚0.4834885334108495 -loss/FRt2/detection_loss‚1.29743451109173 1loss/SRqqt2/detection_loss‚0.6390919760572081 trainer/global_step‚11449 3loss/FBt2/assignment_loss‚0.0005141972914986563 % _timestamp‚1.7842825648261507e+09 /loss/FBt2/detection_loss‚0.7447132182035906 0loss/FRt1/assignment_loss‚1.6203986687113685 0loss/FRt2/assignment_loss‚0.8587288817710425 2loss/FBt1/assignment_loss‚0.010897784305275538 1loss/SRqqt1/detection_loss‚0.9309687828014739 0loss/FBt1/detection_loss‚0.41823853211155015õy–â$‹ /loss/FBt2/detection_loss‚0.7447132182035906 0loss/FRt1/assignment_loss‚1.6203986687113685 0loss/FRt2/assignment_loss‚0.8587288817710425 2loss/FBt1/assignment_loss‚0.010897784305275538 1loss/SRqqt1/detection_loss‚0.9309687828014739 0loss/FBt1/detection_loss‚0.41823853211155015 _step‚943 3loss/SRqqt1/assignment_loss‚0.42400955909383153 3loss/SRqqt2/assignment_loss‚0.49249891165299026 epoch‚28 _runtime‚ 9756.95149455 %loss/total_loss‚7.920983556502408 /loss/FRt1/detection_loss‚0.4834885334108495 -loss/FRt2/detection_loss‚1.29743451109173 1loss/SRqqt2/detection_loss‚0.6390919760572081 trainer/global_step‚11449 3loss/FBt2/assignment_loss‚0.0005141972914986563 % _timestamp‚1.7842825648261507e+09¯ý™zè“ã$‚(:ˆ ÌûçÒž”÷¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i/52ä$‚(:ø ÌûçÒŒ¶Ø¥# proc.memory.rssMB‚ 2821.83984375 cpu‚26.030920783169318 network.sent‚ 1.8171048e+07& disk.nvme0n1.out‚47486.90673828125 disk.sda1.out‚0( disk.nvme0c0n1.out‚47486.90673828125( proc.memory.availableMB‚ 438.83203125 proc.cpu.threads‚17# disk.dm-0.out‚47486.90673828125 disk.nvme0n1.in‚ 467.3203125 disk.drbd2165.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 378.84765625 network.recv‚ 2.601496e+06! disk.nvme0c0n1.in‚ 467.3203125 disk.sda2.in‚0 disk.sda3.out‚ 378.84765625 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.223143577575684) disk./.usagePercent‚36.82854830333772 disk.sda2.out‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚108.64501953125 disk.sda1.in‚0 disk.sda.out‚ 378.84765625 disk.rbd3.in‚0 disk.dm-0.in‚ 467.3203125$ disk./.usageGB‚548.7719841003418 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0$ memory_percent‚97.32158184051514׽淞å$‚(:“ ÛûçÒؒ⢠gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cbÃæ$‚(:ö ÛûçÒÉ““¥ network.sent‚ 1.8234121e+07( disk./.usagePercent‚36.8285444632247$ disk./.usageGB‚548.7719268798828 disk.rbd0.out‚0# proc.memory.rssMB‚ 2821.83984375( disk.nvme0c0n1.out‚47493.91943359375 disk.sda.in‚0 disk.sda.out‚ 379.02734375 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0$ proc.memory.availableMB‚438.8125 network.recv‚ 2.606619e+06 disk.rbd3.out‚0 disk.nvme0n1.in‚ 468.03515625" disk.nvme0c0n1.in‚ 468.03515625 disk.sda3.out‚ 379.02734375 disk.dm-1.out‚ 379.02734375! disk.rbd1.out‚108.70751953125 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚47493.91943359375 disk.rbd3.in‚0& disk.nvme0n1.out‚47493.91943359375 disk.rbd0.in‚0* proc.memory.percent‚17.223143577575684 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 468.03515625 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0$ memory_percent‚97.32170104980469 cpu‚26.030443930858304ÊM½p“ç$‚(:ˆ êûçÒ¡öò¢ gpu.0.gpu‚90.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.132' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}}úè$‚(:ï êûçÒ’­Æ¤ disk.dm-1.in‚0$ disk.nvme0n1.out‚47502.810546875$ disk./.usageGB‚548.7719268798828 disk.rbd3.in‚0 disk.dm-0.in‚ 468.86328125 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚97.3231315612793* proc.memory.percent‚17.223143577575684 cpu‚26.02996896280332 proc.cpu.threads‚17 disk.dm-1.out‚ 379.22265625 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 network.recv‚ 2.609958e+06 disk.sda3.out‚ 379.22265625 disk.nvme0n1.in‚ 468.86328125& disk.nvme0c0n1.out‚47502.810546875 disk.sda2.in‚0& proc.memory.availableMB‚ 438.578125 network.sent‚ 1.823917e+07" disk.nvme0c0n1.in‚ 468.86328125 disk.sda.out‚ 379.22265625# proc.memory.rssMB‚ 2821.83984375( disk./.usagePercent‚36.8285444632247 disk.sda3.in‚0! disk.rbd1.out‚108.70751953125 disk.rbd3.out‚0! disk.dm-0.out‚47502.810546875 disk.sda.in‚0ü$2é$- _wandbruntime‚9796  _runtime‚9796â ãcÉê$à ' lr-AdamW/pg1‚0.00018748159401407938 ' lr-AdamW/pg2‚0.00018748159401407938 trainer/global_step‚11499 $ _timestamp‚1.784282603967911e+09 _runtime‚9796.093504274 _step‚944uzÓNÎë$È ' lr-AdamW/pg1‚0.00018748159401407938 ' lr-AdamW/pg2‚0.00018748159401407938 trainer/global_step‚11499 $ _timestamp‚1.784282603967911e+09 _runtime‚9796.093504274 _step‚944°‹Wû2ì$- _wandbruntime‚9796  _runtime‚9796Q<‹F#í$ûO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12157086189056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1747335530921984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106489836 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1759492975489536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚385000240744448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106499813 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197640344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12157086189056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20143938 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385000222132224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1750489455824497 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1750489098235247 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚397293205903360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12157427528192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91150510 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1759492617111040 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1759492617111040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91150508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44889096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20143941 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24745423 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106489836 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1759492975489536 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1747335530921984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12293076542464 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1738350580338688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24745158 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1738350563299328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106499813 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12138875485809 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91150508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91150510 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197650323 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/reqÛÌÿ´»uested_bytes.large_pool.peak‚ 23025176576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397293317286912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12138534935919 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12157427528192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12138534935919 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12157427528192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106499813 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197640344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12157086189056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20143938 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385000222132224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1750489455824497 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1750489098235247 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚397293205903360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12157427528192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91150510 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1759492617111040 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1759492617111040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91150508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44889096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20143941 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24745423 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106489836 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1759492975489536 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1747335530921984±õüα2ï$- _wandbruntime‚9796  _runtime‚9796è2Mð 'ð$…N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12158098816000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12294093859328 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44892904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106498888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1738497091010560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12139886539643 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91158142 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1747482788990976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197667021 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1738497073971200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12139545989641 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1750636977550203 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1759641229153280 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106498888 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚11499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106508879 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1759641229153280 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1747482771951616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12158098816000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12158440162304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24747221¡Â'ñ$ŠN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1750636977550203 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1759641229153280 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106498888 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚11499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106508879 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1759641229153280 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1747482771951616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12158098816000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12158440162304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24747221 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12158098816000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12294093859328 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44892904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106498888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1738497091010560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12139886539643 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91158142 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1747482788990976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197667021 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1738497073971200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12139545989641 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1766905722472960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106954265 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1754697599442944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106954265 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386671678670848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1745676096892928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12189151422827 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20231191 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12344076493312 trainer/global_step‚11549 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106944288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24848652 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12189491972717 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24848397 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1766905364094464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12207781690880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚949 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1754697599442944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386671660058624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7842826438930328e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20231188 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1745676113932288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91532110 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136ä´ÇG(þ$€P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12208123030016 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1766905364094464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12207781690880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚949 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1754697599442944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚386671660058624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7842826438930328e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20231188 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1745676113932288 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91532110 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1754697582403584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12207781690880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198476396 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399015841644032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1766905722472960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106954265 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1754697599442944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106954265 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386671678670848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1745676096892928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12189151422827 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20231191 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12344076493312 trainer/global_step‚11549 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106944288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24848652 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12189491972717 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24848397 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106953340 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1767053976136704 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45083380 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12345088103424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12209135664128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1767053617751040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106953340 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12209135664128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386705088817152 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1754844840472576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399050288296960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚11549 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1754844823433216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24850447 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1745822624604160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1745822607564800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198503073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20232933 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91539742 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106963331 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12345180867584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399050176920576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106963331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91539740 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 $ _timestamp‚1.784282644503563e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 Q8DeviceStatsMKQ&¢+onitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1767053976136704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91539742 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386705107429376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12190162476549 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1758013127630711 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚9836.629375074 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12208794317824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12208794317824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1758012770041349 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45083654|â}'%‰N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106963331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91539740 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 $ _timestamp‚1.784282644503563e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1767053976136704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91539742 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386705107429376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12190162476549 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1758013127630711 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚9836.629375074 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12208794317824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12208794317824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1758012770041349 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45083654 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106953340 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1767053976136704 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45083380 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12345088103424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12209135664128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1767053617751040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106953340 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12209135664128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386705088817152 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1754844840472576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399050288296960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚11549 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1754844823433216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24850447 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1745822624604160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1745822607564800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198503073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20232933 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91539742 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106963331 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12345180867584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399050176920576¶ÛžNû2‚%- _wandbruntime‚9836  _runtime‚9836ïE4a ƒ%„ 2loss/SRqqt1/assignment_loss‚0.3474063289684982 1loss/SRqqt1/detection_loss‚0.8444504589824937 1loss/SRqqt2/detection_loss‚0.5459313121748312 epoch‚28 2loss/FBt2/assignment_loss‚0.012074871624988296 /loss/FBt2/detection_loss‚0.7945184704937762 % _timestamp‚1.7842826445077145e+09 _step‚951 0loss/FRt1/assignment_loss‚1.4993161268540383 0loss/FRt2/assignment_loss‚0.7409684608000835 2loss/SRqqt2/assignment_loss‚0.4808737408824965 2loss/FBt1/assignment_loss‚0.046382410347374524 /loss/FBt1/detection_loss‚0.2721211649412808 trainer/global_step‚11549 /loss/FRt1/detection_loss‚0.5243943819714094 .loss/FRt2/detection_loss‚1.221138569861957 %loss/total_loss‚7.329576297903227 _runtime‚9836.633361609gèdT„%‰ .loss/FRt2/detection_loss‚1.221138569861957 %loss/total_loss‚7.329576297903227 _runtime‚9836.633361609 2loss/SRqqt1/assignment_loss‚0.3474063289684982 1loss/SRqqt1/detection_loss‚0.8444504589824937 1loss/SRqqt2/detection_loss‚0.5459313121748312 epoch‚28 2loss/FBt2/assignment_loss‚0.012074871624988296 /loss/FBt2/detection_loss‚0.7945184704937762 % _timestamp‚1.7842826445077145e+09 _step‚951 0loss/FRt1/assignment_loss‚1.4993161268540383 0loss/FRt2/assignment_loss‚0.7409684608000835 2loss/SRqqt2/assignment_loss‚0.4808737408824965 2loss/FBt1/assignment_loss‚0.046382410347374524 /loss/FBt1/detection_loss‚0.2721211649412808 trainer/global_step‚11549 /loss/FRt1/detection_loss‚0.5243943819714094·@·ç?‘…%‚(:† —üçÒ€ñ½¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô.ö†%‚(:ë —üçҜſ¤ network.recv‚ 2.62229e+06 disk.rbd1.in‚0.0625! disk.dm-0.out‚47523.267578125 disk.sda.out‚ 379.99609375 proc.cpu.threads‚17 disk.sda3.out‚ 379.99609375( proc.memory.availableMB‚ 438.77734375* proc.memory.percent‚17.223143577575684 network.sent‚ 1.8312269e+07 disk.rbd3.out‚0 disk.nvme0n1.in‚ 470.40625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚47523.267578125! disk.rbd1.out‚108.83251953125 disk.rbd3.in‚0 disk.dm-0.in‚ 470.40625 disk.nvme0c0n1.in‚ 470.40625$ memory_percent‚97.32191562652588$ disk./.usageGB‚548.7716941833496 disk.sda2.out‚0 disk.dm-1.out‚ 379.99609375$ disk.nvme0n1.out‚47523.267578125 disk.sda1.in‚0# proc.memory.rssMB‚ 2821.83984375) disk./.usagePercent‚36.82852884676511 disk.dm-1.in‚0 disk.drbd2165.out‚0 cpu‚26.028593653730976 disk.drbd2165.in‚0â[»‘‡%‚(:† ¦üçÒîŠÌ¢ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆÛžÊûˆ%‚(:ð ¦üçÒšÍŤ# disk.nvme0n1.out‚47532.73046875 disk.rbd3.out‚0# proc.memory.rssMB‚ 2821.83984375 proc.cpu.threads‚17$ disk./.usageGB‚548.7717475891113 disk.nvme0n1.in‚ 471.12109375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 438.98046875* proc.memory.percent‚17.223143577575684 disk.dm-0.in‚ 471.12109375* disk./.usagePercent‚36.828532430870595 disk.drbd2165.out‚0 disk.sda.out‚ 380.1640625 disk.dm-1.out‚ 380.1640625 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 network.sent‚ 1.8375229e+07 disk.sda1.out‚0 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚47532.73046875 disk.dm-1.in‚0 disk.dm-0.out‚47532.73046875$ memory_percent‚97.32067584991455" disk.nvme0c0n1.in‚ 471.12109375 disk.sda3.out‚ 380.1640625! disk.rbd1.out‚108.83251953125 network.recv‚ 2.627633e+06 disk.sda.in‚0 disk.sda2.out‚0 cpu‚26.028095558397972ñ£»“‰%‚(:ˆ µüçÒÙÖ¢ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÍÌðŠ%‚(:ø µüçÒÿâð¤$ disk./.usageGB‚548.7718849182129& disk.nvme0n1.out‚47540.35595703125 disk.drbd2165.in‚0! disk.rbd1.out‚108.95751953125 disk.nvme0n1.in‚ 471.94921875 disk.dm-1.in‚0# disk.dm-0.out‚47540.35595703125# proc.memory.rssMB‚ 2821.83984375 disk.dm-0.in‚ 471.94921875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625# memory_percent‚97.3222017288208 network.recv‚ 2.631197e+06 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 380.3046875 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 380.3046875 disk.rbd3.in‚0 cpu‚26.027622576460935 proc.cpu.threads‚17 network.sent‚ 1.8380266e+07 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 471.94921875( disk.nvme0c0n1.out‚47540.35595703125 disk.sda2.out‚0 disk.sda3.out‚ 380.3046875* disk./.usagePercent‚36.828541647141826 disk.sda1.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 438.73046875* proc.memory.percent‚17.223143577575684Ä•2‹%- _wandbruntime‚9875  _runtime‚9875¶äƒ8ÊŒ%Ä ' lr-AdamW/pg1‚0.00018378175257105207 ' lr-AdamW/pg2‚0.00018378175257105207 trainer/global_step‚11599 % _timestamp‚1.7842826838382201e+09 _runtime‚9875.963804409 _step‚952*ösÏ%É ' lr-AdamW/pg1‚0.00018378175257105207 ' lr-AdamW/pg2‚0.00018378175257105207 trainer/global_step‚11599 % _timestamp‚1.7842826838382201e+09 _runtime‚9875.963804409 _step‚952¸…˜oJ2Ž%- _wandbruntime‚9875  _runtime‚9875cʳ(%ûO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45270628 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24952187 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24951914 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199312448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1762059650924544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20318441 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1753001647525888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1774318469456384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12258477192704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1762059650924544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1774318111077888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12240108459625 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91913710 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199322427 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107408717 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1765241755985513 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12395286620672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12395200140800 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1765241398396263 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚953 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388343097985024´–Ç©]%€P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12258477192704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1762059650924544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1774318111077888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12240108459625 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91913710 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199322427 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107408717 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1765241755985513 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12395286620672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12395200140800 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1765241398396263 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚953 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388343097985024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45270628 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24952187 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24951914 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199312448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1762059650924544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20318441 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚>Ò}©0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1753001647525888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1774318469456384 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199329132 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107407792 trainer/global_step‚11599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20320186 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107417783 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199339125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388376526743552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91921340 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1774466364734464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1753148158197760 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45274176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1753148141158400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91921342 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12259831165952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12241119513459 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24954260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400772755648512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1762206891954176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91921342 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45274446 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚954 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199339125 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12396311183360y²•'“%ŠN P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199329132 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107407792 trainer/global_step‚11599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20320186 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107417783 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199339125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388376526743552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91921340 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1774466364734464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1753148158197760 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45274176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1753148141158400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91921342 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12259831165952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12241119513459 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24954260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400772755648512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1762206891954176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91921342 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45274446 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚954 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199339125 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12396311183360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107407792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12259489819648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169ºñmV€2”%- _wandbruntime‚9876  _runtime‚9876úÜI •%† 0loss/FRt1/assignment_loss‚1.5869502675045153 /loss/FBt1/detection_loss‚0.3771962522309671 2loss/FBt2/assignment_loss‚0.002142670338997947 /loss/FRt1/detection_loss‚0.4966825919079956 _runtime‚9876.579045067 2loss/SRqqt2/assignment_loss‚0.4816648849585207 1loss/FBt1/assignment_loss‚0.01204978417899811 /loss/FRt2/detection_loss‚1.2734467908573426 1loss/SRqqt1/detection_loss‚0.9846025741349396 1loss/SRqqt2/detection_loss‚0.6227072925083506 0loss/FBt2/detection_loss‚0.38201150789179583 trainer/global_step‚11599 $ _timestamp‚1.784282684453719e+09 0loss/FRt2/assignment_loss‚0.7523522606348017 3loss/SRqqt1/assignment_loss‚0.44890642595016894 epoch‚28 _step‚955 &loss/total_loss‚7.4207133030973935_vE$–%‹ 2loss/SRqqt2/assignment_loss‚0.4816648849585207 1loss/FBt1/assignment_loss‚0.01204978417899811 /loss/FRt2/detection_loss‚1.2734467908573426 1loss/SRqqt1/detection_loss‚0.9846025741349396 1loss/SRqqt2/detection_loss‚0.6227072925083506 0loss/FBt2/detection_loss‚0.38201150789179583 trainer/global_step‚11599 $ _timestamp‚1.784282684453719e+09 0loss/FRt2/assignment_loss‚0.7523522606348017 3loss/SRqqt1/assignment_loss‚0.44890642595016894 epoch‚28 _step‚955 &loss/total_loss‚7.4207133030973935 0loss/FRt1/assignment_loss‚1.5869502675045153 /loss/FBt1/detection_loss‚0.3771962522309671 2loss/FBt2/assignment_loss‚0.002142670338997947 /loss/FRt1/detection_loss‚0.4966825919079956 _runtime‚9876.579045067»JÆS|—%‚(:’ ÄüçÒ²¸À¤ gpu.0.gpu‚22.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.811' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.362199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÉrø˜%‚(:í ÄüçҮ߮¦ disk.sda3.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚47550.7392578125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 472.6640625 disk.sda1.out‚0 network.recv‚ 2.634664e+06 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 472.6640625 disk.sda.in‚0 network.sent‚ 1.8385262e+07$ disk./.usageGB‚548.7721366882324 disk.sda2.out‚0 disk.sda3.out‚ 380.578125# proc.memory.rssMB‚ 2821.83984375 cpu‚26.030392328941193 disk.rbd3.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚47550.7392578125 disk.dm-1.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.32217788696289 disk.nvme0n1.in‚ 472.6640625 disk.drbd2165.out‚0 disk.sda.out‚ 380.578125 disk.dm-1.out‚ 380.578125! disk.rbd1.out‚108.95751953125& proc.memory.availableMB‚ 438.734375) disk./.usagePercent‚36.82855854363908 disk.rbd0.in‚0 disk.sda2.in‚0" disk.dm-0.out‚47550.7392578125* proc.memory.percent‚17.223143577575684Ýœ¶è2™%- _wandbruntime‚9893  _runtime‚9893}_g§š%ôz 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7178741693496704 particle/accuracy_3_of_3‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1508333384990692 *jet/accuracy_1_of_2‚0.7820961475372314 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ,jet/accuracy_4_of_4‚0.006769374944269657 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12254916876554489 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_2‚1 % _timestamp‚1.7842827015326517e+09 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8694393634796143 *jet/accuracy_2_of_4‚0.8022790551185608 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891303777694702 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6831466555595398 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5634848475456238 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2216482162475586 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7152907252311707 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'loss/val_total_loss‚7.8883397217464 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20823492109775543 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7295796275138855 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04095751792192459 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05358884483575821 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22020456194877625 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.736967921257019 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3698051869869232 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19726042449474335 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5732894539833069 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5429166555404663 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_2‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3283061981201172 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6419606804847717 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2359440177679062 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 _step‚956 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.032929640263319016 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.018249060958623886 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7550134658813477 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 epoch‚28 8 Purity/’ý*gÓ.2FRt1SRqqt*FBt/FRt_purity‚0.22974088788032532 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24815276265144348 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_3_of_4‚0.06982018053531647 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.48846435546875 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30713436007499695 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6279274821281433 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19399915635585785 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2930982708930969 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2100164145231247 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3308907151222229 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6382437348365784 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6304291486740112 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18776147067546844 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13969776034355164 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27420660853385925 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3716771602630615 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9066470861434937 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2592887282371521 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.626156210899353 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6649234294891357 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4996435046195984 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21288339793682098 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9173845648765564 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7393612861633301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8436802625656128 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988171458244324 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.018513252958655357 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7053457498550415 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25049594044685364 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963498115539551 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6613053679466248 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2082863748073578 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8979337811470032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6979584693908691 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02188521809875965 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6477805376052856 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.01964285597205162 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982649683952332 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6907030344009399 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7775946259498596 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7303626537322998 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6276035308837891 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6864805221557617 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8694393634796143 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6279274821281433 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4990312159061432 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2595781683921814 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04331356659531593 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7352229356765747 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960120320320129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7318821549415588 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8574342131614685 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5342662334442139 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7081255316734314 *jet/accuracy_1_of_3‚0.9218630790710449 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7167892456054688 particle/accuracy_3_of_4‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7970412969589233 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2490769773721695 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02026418223977089 *jet/accuracy_1_of_4‚0.9816306829452515 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚9893.659039826 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5028859972953796 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1524220108985901 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 *jet/accuracy_2_of_3‚0.5637339949607849 +particle/sensitivity‚0.6134307980537415 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4213427007198334 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4771743416786194 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_2‚0.1537674516439438 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22036851942539215 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7393612861633301 particle/accuracy_1_of_4‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.856427788734436 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 +jet/accuracy_3_of_3‚0.06647218763828278 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2400687336921692 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.028406867757439613 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6215739250183105 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6373987197875977 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7722798585891724 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 .particle/accuracy_4_of_4‚0.952063798904419 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019455960020422935 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5852944254875183 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 trainer/global_step‚11599 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7170823812484741 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13505694270133972 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.614878237247467 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6254128217697144 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6022992730140686 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.666874349117279 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25988343358039856 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2490769773721695 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6245232820510864 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02026418223977089 7validation_average_jet_accuracy‚0.37237295508384705 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944258332252502 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 -"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4375 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7956110835075378 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6720709800720215 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7292485237121582 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.359375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7490314841270447 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2245667725801468 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718ÆT =›%ùz 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988171458244324 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.018513252958655357 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7053457498550415 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25049594044685364 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963498115539551 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6613053679466248 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2082863748073578 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8979337811470032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6979584693908691 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02188521809875965 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6477805376052856 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.01964285597205162 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982649683952332 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6907030344009399 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7775946259498596 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7303626537322998 particle/accuracy_1_of_3‚1 'particle/f_score‚0.6276035308837891 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6864805221557617 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8694393634796143 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9537036418914795 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6279274821281433 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4990312159061432 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2595781683921814 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04331356659531593 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7352229356765747 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960120320320129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7318821549415588 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8574342131614685 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5342662334442139 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7081255316734314 *jet/accuracy_1_of_3‚0.9218630790710449 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7167892456054688 particle/accuracy_3_of_4‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7970412969589233 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2490769773721695 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02026418223977089 *jet/accuracy_1_of_4‚0.9816306829452515 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚9893.659039826 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5028859972953796 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1524220108985901 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 *jet/accuracy_2_of_3‚0.5637339949607849 +particle/sensitivity‚0.6134307980537415 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4213427007198334 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4771743416786194 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_2‚0.1537674516439438 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22036851942539215 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7393612861633301 particle/accuracy_1_of_4‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.856427788734436 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 +jet/accuracy_3_of_3‚0.06647218763828278 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2400687336921692 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.028406867757439613 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6215739250183105 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6373987197875977 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7722798585891724 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 .particle/accuracy_4_of_4‚0.952063798904419 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019455960020422935 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5852944254875183 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 trainer/global_step‚11599 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7170823812484741 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13505694270133972 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.614878237247467 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6254128217697144 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6022992730140686 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.666874349117279 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25988343358039856 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2490769773721695 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6245232820510864 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02026418223977089 7validation_average_jet_accuracy‚0.37237295508384705 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944258332252502 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 -"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4375 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7956110835075378 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6720709800720215 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7292485237121582 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.359375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4285714328289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7490314841270447 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2245667725801468 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7178741693496704 particle/accuracy_3_of_3‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1508333384990692 *jet/accuracy_1_of_2‚0.7820961475372314 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ,jet/accuracy_4_of_4‚0.006769374944269657 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12254916876554489 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_2‚1 % _timestamp‚1.7842827015326517e+09 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8694393634796143 *jet/accuracy_2_of_4‚0.8022790551185608 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891303777694702 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6831466555595398 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5634848475456238 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2216482162475586 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7152907252311707 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'loss/val_total_loss‚7.8883397217464 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20823492109775543 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7295796275138855 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04095751792192459 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05358884483575821 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22020456194877625 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.736967921257019 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3698051869869232 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19726042449474335 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5732894539833069 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5429166555404663 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_2‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3283061981201172 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6419606804847717 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2359440177679062 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 _step‚956 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.032929640263319016 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.018249060958623886 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7550134658813477 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 epoch‚28 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22974088788032532 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24815276265144348 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_3_of_4‚0.06982018053531647 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.48846435546875 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30713436007499695 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6279274821281433 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19399915635585785 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2930982708930969 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2100164145231247 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3308907151222229 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6382437348365784 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6304291486740112 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18776147067546844 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13969776034355164 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27420660853385925 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3716771602630615 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9066470861434937 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2592887282371521 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.626156210899353 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6649234294891357 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4996435046195984 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21288339793682098 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9173845648765564 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7393612861633301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8436802625656128 particle/accuracy_4_of_5‚1¼ÖÌu“œ%‚(:ˆ ÓüçÒ±µì¢ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ³¸ô%‚(:õ ÓüçÒ«°è¤ disk.rbd3.out‚0 disk.nvme0n1.in‚ 472.6640625# proc.memory.rssMB‚ 2821.83984375 disk.sda.out‚ 381.0703125( disk.nvme0c0n1.out‚47553.93310546875 disk.rbd3.in‚0# memory_percent‚97.3217248916626 cpu‚26.033471918573525& disk.nvme0n1.out‚47553.93310546875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 381.0703125 disk.dm-0.in‚ 472.6640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0! disk.rbd1.out‚109.02001953125* disk./.usagePercent‚36.828550607405525 network.sent‚ 1.8448538e+07$ disk./.usageGB‚548.7720184326172 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 438.80859375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚47553.93310546875! disk.nvme0c0n1.in‚ 472.6640625 disk.sda2.out‚0 disk.sda3.out‚ 381.0703125* proc.memory.percent‚17.223143577575684 network.recv‚ 2.640421e+06Ü¢<›ž%‚(:’ âüçÒœÄߣ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.184' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.63680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l²8ÏþŸ%‚(:ó âüçÒòò÷¤$ memory_percent‚97.32568264007568 proc.cpu.threads‚17) disk./.usagePercent‚36.82848942160484 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 381.2578125! disk.rbd1.out‚109.02001953125* proc.memory.percent‚17.223143577575684 disk.rbd0.in‚0 disk.sda.out‚ 381.2578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2821.83984375 cpu‚26.03299440322482 network.recv‚ 2.64527e+06& disk.nvme0n1.out‚47563.81884765625 disk.sda.in‚0# disk.dm-0.out‚47563.81884765625 disk.sda1.out‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 473.4921875 disk.rbd3.out‚0 disk.dm-0.in‚ 473.4921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚47563.81884765625( proc.memory.availableMB‚ 438.16015625$ disk./.usageGB‚548.7711067199707 disk.rbd3.in‚0 disk.dm-1.out‚ 381.2578125 network.sent‚ 1.8505995e+07 disk.nvme0n1.in‚ 473.4921875;Z$‰› %‚(: ñüçÒÈ‹ž¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I’•Î¡%‚(:ø ñüçÒ٭إ disk.drbd2165.out‚0# proc.memory.rssMB‚ 2821.83984375 cpu‚26.032564557986014 disk.sda.out‚ 381.46484375 disk.dm-1.out‚ 381.46484375" disk.dm-0.out‚47572.0419921875 disk.sda1.out‚0 disk.sda3.out‚ 381.46484375 network.recv‚ 2.648746e+06) disk./.usagePercent‚36.82849274970278' disk.nvme0c0n1.out‚47572.0419921875 disk.rbd3.out‚0 disk.sda1.in‚0$ memory_percent‚97.32716083526611 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 474.20703125 disk.rbd3.in‚0 disk.sda.in‚0! disk.rbd1.out‚109.08251953125 disk.nvme0n1.in‚ 474.20703125 network.sent‚ 1.8510892e+07 disk.rbd1.in‚0.0625% disk.nvme0n1.out‚47572.0419921875 disk.drbd2165.in‚0 disk.sda2.out‚0$ disk./.usageGB‚548.7711563110352 disk.rbd0.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 437.91796875 disk.sda3.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.223143577575684 disk.rbd0.in‚0 disk.dm-0.in‚ 474.20703125×*N2¢%- _wandbruntime‚9932  _runtime‚9932–í¬æÊ£%Ä _step‚957 ' lr-AdamW/pg1‚0.00018193683550263545 ' lr-AdamW/pg2‚0.00018193683550263545 trainer/global_step‚11649 % _timestamp‚1.7842827406495476e+09 _runtime‚9932.776368962O»œ·Ï¤%É _runtime‚9932.776368962 _step‚957 ' lr-AdamW/pg1‚0.00018193683550263545 ' lr-AdamW/pg2‚0.00018193683550263545 trainer/global_step‚11649 % _timestamp‚1.7842827406495476e+09½,À@T2¥%- _wandbruntime‚9932  _runtime‚9932¦w[ÿŠ¦%úO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107933491 WGDeviceStatsMonitor.on_train_batch_start/allocnïñÄv'ated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12464608377856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1771351252017152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390030448011264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12308104121872 _runtime‚9932.776445547 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200349467 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107923514 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20408211 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25086789 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1774543931416080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1762235844333568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1783678157917184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1771351252017152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402495075001344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11649 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92415976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12308444671762 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200339488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92415974 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12327247239168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390030466623488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12464530286592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12327247239168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45494731 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1783678516295680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92415974 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1783678157917184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12326905900032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1774544289005330 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169©0»(§%ÿO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11649 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92415976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12308444671762 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200339488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92415974 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12327247239168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390030466623488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12464530286592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12327247239168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45494731 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1783678516295680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92415974 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1783678157917184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12326905900032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1774544289005330 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107933491 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12464608377856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1771351252017152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390030448011264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12308104121872 _runtime‚9932.776445547 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200349467 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107923514 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20408211 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25086789 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1774543931416080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1762235844333568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1783678157917184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1771351252017152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402495075001344¾~?l82¨%- _wandbruntime‚9933  _runtime‚9933Wl[ '©%…N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25088866 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1771498510086144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107932566 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402529531081216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1762382337966080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚959 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1771498510086144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12465551323648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚9933.389173494 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20409953 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12327918526976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1771498493046784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92423608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45498564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1774691453141674 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45498822 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12309115175594 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390063895382016 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1774691810731036 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200366165 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200356172 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107942557 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12328259873280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12328259873280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20409956 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1783826411573760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1783826769959424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200356172 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200366165 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92423608 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12309115175594 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390063895382016 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1774691810731036 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200366165 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200356172 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107942557 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12328259873280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12328259873280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20409956 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1783826411573760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1783826769959424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200356172 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200366165 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92423608 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12327918526976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1771498493046784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92423608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45498564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77¿ Hõr2«%- _wandbruntime‚9933  _runtime‚9933òn&¬¬%ˆ trainer/global_step‚11649 _step‚960 3loss/SRqqt1/assignment_loss‚0.37142508232988264 3loss/SRqqt2/assignment_loss‚0.45057387344161115 %loss/total_loss‚7.397603758388295 _runtime‚9933.390380632 0loss/FRt1/assignment_loss‚1.5155295159859272 0loss/FRt2/assignment_loss‚0.8023802338914479 2loss/FBt2/assignment_loss‚0.008534425778085535 /loss/FRt2/detection_loss‚1.2513917328899706 1loss/SRqqt2/detection_loss‚0.6440980994444717 0loss/FBt1/detection_loss‚0.32524239366353286 epoch‚29 % _timestamp‚1.7842827412650359e+09 /loss/FRt1/detection_loss‚0.5240884531833173 1loss/SRqqt1/detection_loss‚0.9352612693392788 /loss/FBt2/detection_loss‚0.5638484344978271 2loss/FBt1/assignment_loss‚0.005230243942941805UØêF­% epoch‚29 % _timestamp‚1.7842827412650359e+09 /loss/FRt1/detection_loss‚0.5240884531833173 1loss/SRqqt1/detection_loss‚0.9352612693392788 /loss/FBt2/detection_loss‚0.5638484344978271 2loss/FBt1/assignment_loss‚0.005230243942941805 trainer/global_step‚11649 _step‚960 3loss/SRqqt1/assignment_loss‚0.37142508232988264 3loss/SRqqt2/assignment_loss‚0.45057387344161115 %loss/total_loss‚7.397603758388295 _runtime‚9933.390380632 0loss/FRt1/assignment_loss‚1.5155295159859272 0loss/FRt2/assignment_loss‚0.8023802338914479 2loss/FBt2/assignment_loss‚0.008534425778085535 /loss/FRt2/detection_loss‚1.2513917328899706 1loss/SRqqt2/detection_loss‚0.6440980994444717 0loss/FBt1/detection_loss‚0.32524239366353286ÀXf»@®%‚(:„ €ýçÒ¥À–¥ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä€Yú¯%‚(:ï €ýçÒÃÌÕ¦* proc.memory.percent‚17.223143577575684 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0) disk./.usagePercent‚36.82847918130347 disk.rbd3.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17! disk.dm-0.out‚47582.537109375 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 475.03515625 disk.rbd1.in‚0.0625! disk.rbd1.out‚109.08251953125 disk.dm-0.in‚ 475.03515625 disk.nvme0n1.in‚ 475.03515625 disk.sda.out‚ 381.703125( proc.memory.availableMB‚ 438.12109375$ disk./.usageGB‚548.7709541320801 disk.dm-1.out‚ 381.703125$ disk.nvme0n1.out‚47582.537109375 disk.sda3.out‚ 381.703125 cpu‚26.032109941532788 network.sent‚ 1.8528942e+07 network.recv‚ 2.651931e+06 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.in‚0& disk.nvme0c0n1.out‚47582.537109375 disk.sda2.in‚0$ memory_percent‚97.32592105865479# proc.memory.rssMB‚ 2821.83984375¿ìÈÍ’°%‚(:‡ ýçÒƞ¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.045' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.809 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`‚K±%‚(:û ýçҥȧ¥ disk.sda2.out‚0 cpu‚26.031644518393172 network.sent‚ 1.8579046e+07& disk.nvme0n1.out‚47586.02978515625 disk.dm-1.in‚0 disk.dm-1.out‚ 382.05078125 disk.rbd1.in‚0.0625 network.recv‚ 2.657428e+06) disk./.usagePercent‚36.82852142254662 disk.nvme0n1.in‚ 475.03515625 disk.sda2.in‚0 disk.sda3.out‚ 382.05078125( proc.memory.availableMB‚ 437.93359375 disk.rbd3.out‚0 disk.dm-0.in‚ 475.03515625 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 475.03515625 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚47586.02978515625 disk.sda.out‚ 382.05078125! disk.rbd1.out‚109.14501953125* proc.memory.percent‚17.223143577575684 proc.cpu.threads‚17# disk.dm-0.out‚47586.02978515625 disk.sda1.out‚0$ memory_percent‚97.32706546783447# proc.memory.rssMB‚ 2821.83984375$ disk./.usageGB‚548.7715835571289 disk.drbd2165.in‚0 disk.rbd0.in‚0 2²%- _wandbruntime‚9972  _runtime‚99725™ÐMʳ%Ä % _timestamp‚1.7842827805569403e+09 _runtime‚9972.682543337 _step‚961 ' lr-AdamW/pg1‚0.00018009541226931525 ' lr-AdamW/pg2‚0.00018009541226931525 trainer/global_step‚11699f) £Ï´%É ' lr-AdamW/pg1‚0.00018009541226931525 ' lr-AdamW/pg2‚0.00018009541226931525 trainer/global_step‚11699 % _timestamp‚1.7842827805569403e+09 _runtime‚9972.682543337 _step‚961ÁîÝjº2µ%- _wandbruntime‚9972  _runtime‚9972â›V`(¶%ûO aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1769561377927168 trainer/global_step‚11699 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1791091263279104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12377601401856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20495458 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12515679351808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12377601401856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1778713303498752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108377966 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1778713320538112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45685236 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25189778 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404217672478720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92797576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12377942740992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1791090904900608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9972.683786113 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚962 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92797574 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12377942740992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1791091263279104 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚391701885937664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12358720608780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92797574 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108387943 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108387943 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201175540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92797576 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12359061158670 % _timestamp‚1.7842827805580056e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25190057 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108377966 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45685518 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1791090904900608 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1778713320538112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12515767928832 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1781920439085838PÀýA&·%€P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1791090904900608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9972.683786113 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚962 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92797574 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12377942740992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1791091263279104 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚391701885937664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12358720608780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92797574 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108387943 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108387943 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201175540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92797576 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12359061158670 % _timestamp‚1.7842827805580056e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25190057 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108377966 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45685518 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1791090904900608 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1778713320538112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12515767928832 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1781920439085838 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1769561377927168 trainer/global_step‚11699 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1791091263279104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12377601401856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20495458 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12515679351808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12377601401856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1778713303498752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108377966 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1778713320538112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45685236 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25189778 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404217672478720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batchS­AÅ_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92797576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12377942740992¼"FÖ2¸%- _wandbruntime‚9973  _runtime‚9973¦Pþ± '¹%…N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92805206 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1778860561567744 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201202217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11699 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1791239158557184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108387018 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1778860561567744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12378955375104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201202217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45689292 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1791239516942848 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12378614028800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404252012940800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12516698244608 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1778860544528384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1782067603222182 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108397009 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1778860544528384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1782067960811544 _runtime‚9973.296016604 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108387018 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20497206 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12378955375104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92805208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12360072212504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842827811702178e+09 _step‚963 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45689017 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20497203 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201192224 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45689017 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20497203 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201192224 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92805206 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1778860561567744 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201202217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11699 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1791239158557184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108387018 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1778860561567744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12378955375104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201202217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45689292 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1791239516942848 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12378614028800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404252012940800 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12516698244608 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1778860544528384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1782067603222182 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108397009 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1778860544528384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1782067960811544 _runtime‚9973.296016604 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108387018 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20497206 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12378955375104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92805208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ÃÈUßœ2»%- _wandbruntime‚9973  _runtime‚9973÷÷XÞ ¼%† 0loss/FRt2/assignment_loss‚0.8009009019170009 2loss/SRqqt2/assignment_loss‚0.4422295198336705 3loss/FBt2/assignment_loss‚6.429024924840358e-05 0loss/FRt1/detection_loss‚0.49857458031288415 /loss/FRt2/detection_loss‚1.1653405812640287 trainer/global_step‚11699 2loss/SRqqt1/assignment_loss‚0.3835991680512572 1loss/SRqqt1/detection_loss‚0.9601684835141719 epoch‚29 % _timestamp‚1.7842827811733923e+09 _runtime‚9973.298730036 /loss/FBt2/detection_loss‚0.5941790518506119 /loss/FRt1/assignment_loss‚1.551169423018088 1loss/SRqqt2/detection_loss‚0.5956424931986729 /loss/FBt1/detection_loss‚0.4430438141721157 2loss/FBt1/assignment_loss‚0.028458333714081272 _step‚964 %loss/total_loss‚7.463370641095832‰š˜ ½%‹ % _timestamp‚1.7842827811733923e+09 _runtime‚9973.298730036 /loss/FBt2/detection_loss‚0.5941790518506119 /loss/FRt1/assignment_loss‚1.551169423018088 1loss/SRqqt2/detection_loss‚0.5956424931986729 /loss/FBt1/detection_loss‚0.4430438141721157 2loss/FBt1/assignment_loss‚0.028458333714081272 _step‚964 %loss/total_loss‚7.463370641095832 0loss/FRt2/assignment_loss‚0.8009009019170009 2loss/SRqqt2/assignment_loss‚0.4422295198336705 3loss/FBt2/assignment_loss‚6.429024924840358e-05 0loss/FRt1/detection_loss‚0.49857458031288415 /loss/FRt2/detection_loss‚1.1653405812640287 trainer/global_step‚11699 2loss/SRqqt1/assignment_loss‚0.3835991680512572 1loss/SRqqt1/detection_loss‚0.9601684835141719 epoch‚29Äk±ï¾%‚(:ä žýçÒëÄ¥$ memory_percent‚97.32661247253418' proc.memory.availableMB‚ 438.0078125 cpu‚26.03114681061253 disk.sda3.in‚0) disk./.usagePercent‚36.82851553437334 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0$ disk./.usageGB‚548.7714958190918 disk.nvme0n1.in‚475.75& disk.nvme0n1.out‚47594.98095703125 disk.dm-0.in‚475.75 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 382.4453125* proc.memory.percent‚17.223143577575684 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚109.14501953125 disk.rbd3.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2821.83984375( disk.nvme0c0n1.out‚47594.98095703125 disk.sda.out‚ 382.4453125 disk.sda3.out‚ 382.4453125 network.sent‚ 1.8583869e+07 network.recv‚ 2.660023e+06# disk.dm-0.out‚47594.98095703125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚475.75š³ì‘¿%‚(:† žýçÒ¡Üä¸ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚193.66' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0caÃò“À%‚(:ˆ ­ýçÒùÃТ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!à(âÁ%‚(:× ­ýçÒœ®Š¥ disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚382.6875! disk.dm-0.out‚47602.306640625* proc.memory.percent‚17.223143577575684 cpu‚26.03067916919566 disk.dm-0.in‚ 476.578125 network.sent‚ 1.8646116e+07$ disk./.usageGB‚548.7725677490234 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚382.6875! disk.rbd1.out‚109.20751953125 network.recv‚ 2.665035e+06* disk./.usagePercent‚36.828587472490455 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚47602.306640625 disk.dm-1.out‚382.6875 disk.rbd3.out‚0! memory_percent‚97.32666015625 disk.nvme0c0n1.in‚ 476.578125 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2821.83984375 disk.nvme0n1.in‚ 476.578125$ disk.nvme0n1.out‚47602.306640625 disk.sda.in‚0 disk.rbd1.in‚0.0625 proc.memory.availableMB‚438( n“Â%‚(:ˆ ¼ýçÒ¤Ú¢ gpu.0.gpu‚93.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wwYŸúÃ%‚(:ï ¼ýçÒöÎñ¤ disk.dm-0.in‚ 477.29296875 disk.rbd0.out‚0$ memory_percent‚97.32522964477539 cpu‚26.030211548559453 network.sent‚ 1.8651246e+07 disk.sda2.out‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚47612.72314453125 disk.sda.out‚ 383.09375& proc.memory.availableMB‚ 438.234375 proc.cpu.threads‚17 network.recv‚ 2.668436e+06# disk./.usageGB‚548.767333984375 disk.sda3.in‚0 disk.dm-1.out‚ 383.09375 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.223143577575684 disk.sda3.out‚ 383.09375 disk.rbd3.in‚0# disk.dm-0.out‚47612.72314453125 disk.nvme0n1.in‚ 477.29296875 disk.sda1.in‚0 disk.drbd2165.in‚0! disk.rbd1.out‚109.20751953125 disk.sda1.out‚0( disk.nvme0c0n1.out‚47612.72314453125 disk.sda.in‚0) disk./.usagePercent‚36.82823623015348 disk.sda2.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 477.29296875# proc.memory.rssMB‚ 2821.83984375(¯ìF4Ä%/ _wandbruntime‚10012  _runtime‚10012…c"ÈÅ% & lr-AdamW/pg2‚0.0001782576011422646 trainer/global_step‚11749 % _timestamp‚1.7842828202268393e+09 _runtime‚10012.35245215 _step‚965 & lr-AdamW/pg1‚0.0001782576011422646r†#¡ÍÆ%Ç _runtime‚10012.35245215 _step‚965 & lr-AdamW/pg1‚0.0001782576011422646 & lr-AdamW/pg2‚0.0001782576011422646 trainer/global_step‚11749 % _timestamp‚1.7842828202268393e+09ÅܽÞÃ4Ç%/ _wandbruntime‚10012  _runtime‚10012þܳ¢È%üO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93179174 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12428638242816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93179174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1776886894481408 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12409677645578 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20582708 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1786075372019712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202021571 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393373342476288 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12409337095688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202011592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚11749 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45875889 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1786075372019712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25292904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12567004295168 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1776886911520768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12428296903680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1789296231577096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1798503651884032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405940241679360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚966 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393373323864064 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1798504010262528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108842395 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108832418 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108842395 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1786075354980352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12428296903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚u®`0 _runtime‚10012.353716767 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93179176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405940346771456 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12566917815296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1789296589166346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1798504010262528 % _timestamp‚1.7842828202279088e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12428638242816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25293178 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1798503651884032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45875612 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20582711 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108832418 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93179176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438š%@Ó(É%P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1786075372019712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202021571 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393373342476288 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12409337095688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202011592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚11749 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45875889 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1786075372019712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25292904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12567004295168 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1776886911520768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12428296903680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1789296231577096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1798503651884032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405940241679360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚966 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393373323864064 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1798504010262528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108842395 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108832418 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108842395 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1786075354980352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12428296903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚10012.353716767 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93179176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405940346771456 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12566917815296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1789296589166346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1798504010262528 % _timestamp‚1.7842828202279088e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12428638242816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25293178 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1798503651884032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45875612 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20582711 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108832418 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93179176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93179174 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12428638242816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93179174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1776886894481408 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12409677645578 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20582708 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250ÆT¼f˜4Ê%/ _wandbruntime‚10013  _runtime‚10013ü7•M 'Ë%…N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202028276 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25294945 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1798652263926272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784282821028585e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1786222596009984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108841470 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93186806 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12567933652480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202038269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1777033405153280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1798651905540608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93186808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1789443753302690 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚393406771234816 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1789444110892052 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚967 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1786222613049344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1786222613049344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11749 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45879655 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12429309530624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12410348149410 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20584456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25295199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1786222596009984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12410688699412 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108841470 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20584453 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚393406752622592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202028276 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93186808 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93186806 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12429650876928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45879398 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976J:Â7Ì%ŠN QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚967 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1786222613049344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1786222613049344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11749 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45879655 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12429309530624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12410348149410 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20584456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25295199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1786222596009984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12410688699412 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108841470 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20584453 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚393406752622592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202028276 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93186808 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93186806 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12429650876928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45879398 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202028276 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25294945 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1798652263926272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784282821028585e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1786222596009984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 A8DeviceStatsMonitor.on_train_batch_end/active.all.cîßgÂÙurrent‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108841470 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93186806 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12567933652480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202038269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1777033405153280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1798651905540608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93186808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1789443753302690 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚393406771234816 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1789444110892052 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269ÇnlMƒ4Í%/ _wandbruntime‚10013  _runtime‚10013%nvÎ%ˆ /loss/FBt1/detection_loss‚0.3206425386474839 _step‚968 /loss/FBt2/detection_loss‚0.7190498650665037 trainer/global_step‚11749 0loss/FRt1/assignment_loss‚1.5447202879254536 2loss/SRqqt1/assignment_loss‚0.3897513883973583 4loss/FBt2/assignment_loss‚4.4440948561084675e-05 1loss/SRqqt2/detection_loss‚0.5736725080362161 %loss/total_loss‚7.671048848021082 0loss/FRt2/assignment_loss‚0.9441678281309027 epoch‚29 % _timestamp‚1.7842828210319924e+09 _runtime‚10013.157352229 2loss/SRqqt2/assignment_loss‚0.4849143276720077 2loss/FBt1/assignment_loss‚0.003937537797271666 /loss/FRt1/detection_loss‚0.5039424612720811 /loss/FRt2/detection_loss‚1.2331060952923454 1loss/SRqqt1/detection_loss‚0.9530995688348959‰kd®Ï% 2loss/SRqqt2/assignment_loss‚0.4849143276720077 2loss/FBt1/assignment_loss‚0.003937537797271666 /loss/FRt1/detection_loss‚0.5039424612720811 /loss/FRt2/detection_loss‚1.2331060952923454 1loss/SRqqt1/detection_loss‚0.9530995688348959 /loss/FBt1/detection_loss‚0.3206425386474839 _step‚968 /loss/FBt2/detection_loss‚0.7190498650665037 trainer/global_step‚11749 0loss/FRt1/assignment_loss‚1.5447202879254536 2loss/SRqqt1/assignment_loss‚0.3897513883973583 4loss/FBt2/assignment_loss‚4.4440948561084675e-05 1loss/SRqqt2/detection_loss‚0.5736725080362161 %loss/total_loss‚7.671048848021082 0loss/FRt2/assignment_loss‚0.9441678281309027 epoch‚29 % _timestamp‚1.7842828210319924e+09 _runtime‚10013.157352229Ȳ±à‘Ð%‚(:† ËýçÒžÿË£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§åß´îÑ%‚(:ã ËýçÒ¸‰þ¤ disk.dm-1.out‚ 383.34375 disk.rbd1.in‚0.0625 disk.sda1.in‚0# proc.memory.rssMB‚ 2821.83984375 network.sent‚ 1.8656223e+07 disk.rbd3.out‚0 disk.dm-0.in‚ 478.1171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0! memory_percent‚97.32666015625 cpu‚26.02976859486012 disk.nvme0n1.in‚ 478.1171875 disk.rbd0.in‚0! disk.rbd1.out‚109.33251953125# disk.dm-0.out‚47621.03173828125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚47621.03173828125! disk.nvme0c0n1.in‚ 478.1171875( disk.nvme0c0n1.out‚47621.03173828125* disk./.usagePercent‚36.828257222771285$ disk./.usageGB‚548.7676467895508 disk.sda.in‚0 disk.sda.out‚ 383.34375 disk.sda3.out‚ 383.34375 network.recv‚ 2.672024e+06 disk.sda3.in‚0 disk.rbd0.out‚0 proc.memory.availableMB‚438* proc.memory.percent‚17.223143577575684“ã/o‘Ò%‚(:† ÚýçÒ¹Ùî£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.866' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.1732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ÿÚêÓ%‚(:ß ÚýçÒ…Ô¨¥ cpu‚26.029350554703797 disk.sda3.out‚383.5! disk.nvme0c0n1.in‚ 478.1171875 disk.rbd3.in‚0 disk.dm-0.in‚ 478.1171875 disk.sda1.in‚0 disk.sda.out‚383.5# proc.memory.rssMB‚ 2821.83984375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 478.1171875 disk.drbd2165.in‚0) disk./.usagePercent‚36.82825927083156 disk.sda3.in‚0* proc.memory.percent‚17.223143577575684$ disk./.usageGB‚548.7676773071289! disk.rbd1.out‚109.33251953125" disk.dm-0.out‚47627.5595703125% disk.nvme0n1.out‚47627.5595703125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚383.5 disk.rbd1.in‚0.0625$ memory_percent‚97.32565879821777' proc.memory.availableMB‚ 438.1640625 network.sent‚ 1.8719357e+07 network.recv‚ 2.677006e+06 disk.rbd3.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚47627.5595703125 disk.sda2.out‚0 Æ®ÒÔ%‚(:’ éýçÒŒ†®¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.30499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ fÕ%‚(:õ éýçÒ€¦¥ disk.sda.in‚0 disk.dm-0.in‚ 478.83203125# proc.memory.rssMB‚ 2821.83984375 network.recv‚ 2.68039e+06$ disk./.usageGB‚548.7726745605469 disk.sda3.out‚ 383.69140625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 478.83203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 478.83203125' disk.nvme0c0n1.out‚47634.5498046875* proc.memory.percent‚17.223143577575684 cpu‚26.02888708766074" disk.dm-0.out‚47634.5498046875% disk.nvme0n1.out‚47634.5498046875 disk.drbd2165.out‚0 disk.sda.out‚ 383.69140625 disk.sda2.out‚0$ memory_percent‚97.32718467712402' proc.memory.availableMB‚ 437.9140625 proc.cpu.threads‚17) disk./.usagePercent‚36.82859464070141 disk.dm-1.out‚ 383.69140625 network.sent‚ 1.8724279e+07 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚109.39501953125°Ù$4Ö%/ _wandbruntime‚10052  _runtime‚10052é•Ê×%Ä ' lr-AdamW/pg1‚0.00017642352016065805 ' lr-AdamW/pg2‚0.00017642352016065805 trainer/global_step‚11799 $ _timestamp‚1.784282860091603e+09 _runtime‚10052.217318763 _step‚969å#',ÏØ%É _step‚969 ' lr-AdamW/pg1‚0.00017642352016065805 ' lr-AdamW/pg2‚0.00017642352016065805 trainer/global_step‚11799 $ _timestamp‚1.784282860091603e+09 _runtime‚10052.217318763ÉWJ°Ø4Ù%/ _wandbruntime‚10052  _runtime‚10052 Ç (Ú%üO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12478992405504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12618160527872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1793437406461952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25396272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407663029507584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46066233 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20669958 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395044761790464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842828600927439e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202847644 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1805916398867456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1796672381657604 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚970 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12479333744640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395044780402688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109286870 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93560774 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93560776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46065960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12618249104896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1784212445114368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1793437423501312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1805916398867456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407662922318336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109286870 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109296847 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12460294132486 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12459953582596 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1784212428075008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93560776 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93560774 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1805916757245952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1793437423501312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚11799ªÒ€½(Û%P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1793437423501312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1805916398867456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407662922318336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109286870 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109296847 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12460294132486 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12459953582596 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1784212428075008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93560776 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93560774 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1805916757245952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1793437423501312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚11799 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12478992405504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12618160527872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1793437406461952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25396272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407663029507584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46066233 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20669958 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395044761790464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842828600927439e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202847644 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1805916398867456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1796672381657604 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚970 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12479333744640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395044780402688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109286870 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93560774 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93560776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46065960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12618249104896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1784212445114368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976Ê {—Œ4Ü%/ _wandbruntime‚10052  _runtime‚10052x*Ý%†N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46070088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20671703 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1784358938746880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚971 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202864328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407697481026048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93568406 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109295922 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1793584664530944 trainer/global_step‚11799 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1793584664530944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395078190548992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93568408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109295922 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1796820260972560 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202874321 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25398124 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12480346378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1806064652524032 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46069827 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 % _timestamp‚1.7842828607025952e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12480346378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚10052.828502484 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1806065010909696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1793584647491584UZÀr'Þ%‹N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _step‚971 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202864328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407697481026048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93568406 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109295922 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1793584664530944 trainer/global_step‚11799 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1793584664530944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395078190548992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93568408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109295922 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1796820260972560 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202874321 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25398124 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12480346378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1806064652524032 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46069827 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 % _timestamp‚1.7842828607025952e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12480346378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚10052.828502484 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1806065010909696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1793584647491584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46070088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20671703 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1784358938746880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ëþi¥ 4ß%/ _wandbruntime‚10052  _runtime‚10052Qb à%‡ 2loss/FBt1/assignment_loss‚0.002107536172543543 1loss/SRqqt1/detection_loss‚0.7775462682379617 trainer/global_step‚11799 _runtime‚10052.831718475 0loss/FRt2/assignment_loss‚0.9358043085046491 3loss/SRqqt1/assignment_loss‚0.49088876552220123 /loss/FRt2/detection_loss‚1.2394078792663932 1loss/SRqqt2/detection_loss‚0.6184077671210291 % _timestamp‚1.7842828607063773e+09 _step‚972 2loss/FBt2/assignment_loss‚0.022525158362090427 /loss/FRt1/detection_loss‚0.5205872728699054 0loss/FBt1/detection_loss‚0.43610675332520793 epoch‚29 /loss/FRt1/assignment_loss‚1.528237039906638 2loss/SRqqt2/assignment_loss‚0.5052688349559558 /loss/FBt2/detection_loss‚0.8173757039757189 %loss/total_loss‚7.894263288220294ÜeÍ”á%Œ /loss/FRt1/assignment_loss‚1.528237039906638 2loss/SRqqt2/assignment_loss‚0.5052688349559558 /loss/FBt2/detection_loss‚0.8173757039757189 %loss/total_loss‚7.894263288220294 2loss/FBt1/assignment_loss‚0.002107536172543543 1loss/SRqqt1/detection_loss‚0.7775462682379617 trainer/global_step‚11799 _runtime‚10052.831718475 0loss/FRt2/assignment_loss‚0.9358043085046491 3loss/SRqqt1/assignment_loss‚0.49088876552220123 /loss/FRt2/detection_loss‚1.2394078792663932 1loss/SRqqt2/detection_loss‚0.6184077671210291 % _timestamp‚1.7842828607063773e+09 _step‚972 2loss/FBt2/assignment_loss‚0.022525158362090427 /loss/FRt1/detection_loss‚0.5205872728699054 0loss/FBt1/detection_loss‚0.43610675332520793 epoch‚29Ì‘dª†â%‚(:… øýçÒèº÷¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.23 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k Wã%‚(:ü øýçÒ„æñ¤ disk.dm-0.in‚ 479.66015625 disk.sda1.out‚0 disk.sda.in‚0 network.recv‚ 2.683517e+06* disk./.usagePercent‚36.828601296897304 disk.rbd3.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚47643.95458984375 disk.sda.out‚ 384.02734375 network.sent‚ 1.8729261e+07 disk.rbd3.out‚0 disk.sda2.out‚0$ memory_percent‚97.32725620269775( proc.memory.availableMB‚ 437.90234375# proc.memory.rssMB‚ 2821.83984375* proc.memory.percent‚17.223143577575684 cpu‚26.028449746824183$ disk./.usageGB‚548.7727737426758 disk.rbd1.in‚0.0625 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 384.02734375 disk.dm-1.in‚0 disk.dm-1.out‚ 384.02734375# disk.dm-0.out‚47643.95458984375 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 479.66015625 proc.cpu.threads‚17! disk.rbd1.out‚109.39501953125 disk.sda2.in‚0 disk.nvme0n1.in‚ 479.66015625& disk.nvme0n1.out‚47643.954589843753"“|“ä%‚(:ˆ ‡þçÒùÙ£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚268.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ml îå%‚(:ã ‡þçÒýÙ…¥ proc.cpu.threads‚17 disk.nvme0c0n1.in‚480.375( disk.nvme0c0n1.out‚47650.35400390625 disk.sda2.in‚0 disk.dm-1.out‚ 384.296875# disk.dm-0.out‚47650.35400390625 cpu‚26.02796233840043$ disk./.usageGB‚548.7728233337402 disk.dm-0.in‚480.375# proc.memory.rssMB‚ 2821.83984375 disk.drbd2165.in‚0 network.recv‚ 2.688807e+06) disk./.usagePercent‚36.82860462499525 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.drbd2165.out‚0& proc.memory.availableMB‚ 438.265625* proc.memory.percent‚17.223143577575684 disk.rbd3.in‚0& disk.nvme0n1.out‚47650.35400390625$ memory_percent‚97.32503890991211 network.sent‚ 1.8792409e+07 disk.sda3.out‚ 384.296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 384.296875 disk.dm-1.in‚0! disk.rbd1.out‚109.45751953125 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚480.375 disk.sda1.out‚0±á’S4æ%/ _wandbruntime‚10092  _runtime‚10092`ÅžËç%Å _runtime‚10092.140400919 _step‚973 ' lr-AdamW/pg1‚0.00017459328712408976 ' lr-AdamW/pg2‚0.00017459328712408976 trainer/global_step‚11849 % _timestamp‚1.7842829000136435e+09Xh³Ðè%Ê ' lr-AdamW/pg1‚0.00017459328712408976 ' lr-AdamW/pg2‚0.00017459328712408976 trainer/global_step‚11849 % _timestamp‚1.7842829000136435e+09 _runtime‚10092.140400919 _step‚973ÍV²û¯4é%/ _wandbruntime‚10092  _runtime‚10092@yI(ê%ûO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25499324 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12669308550656 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1791537961668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12529687907328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12530029246464 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46256261 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1804048889327362 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203683696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1813329504229376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1791537978707968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚974 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1804048531738112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12510570069504 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93942374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚396716199716864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12510910619394 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93942376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚396716218329088 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12530029246464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20757211 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12669403419136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1813329145850880 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1800799474982912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚11849 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109751299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109741322 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1800799457943552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12529687907328 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1813329504229376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1800799474982912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109741322 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409385621748224¶°ˆ€Šë%€P C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1800799474982912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109741322 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409385621748224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25499324 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12669308550656 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_‚›¨¢|#pool.freed‚1791537961668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12529687907328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12530029246464 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46256261 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1804048889327362 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203683696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1813329504229376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1791537978707968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚974 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1804048531738112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12510570069504 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93942374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚396716199716864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12510910619394 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93942376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚396716218329088 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12530029246464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20757211 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12669403419136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1813329145850880 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1800799474982912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚11849 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109751299 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109741322 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1800799457943552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12529687907328 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1813329504229376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Î ƒÜû4ì%/ _wandbruntime‚10092  _runtime‚10092[o@I 'í%†N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203700380 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1800946698973184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396749628475392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚11849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203710373 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46260329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1791684472340480 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7842829006278985e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396749647087616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203710373 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409419962107904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93950008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1813477757893120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46260042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25501089 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12511581123226 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12511921673228 _runtime‚10092.753695206 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109750374 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1800946716012544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1813477757893120 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1813477399507456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12530700534272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20758953 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203700380 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25501373 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚975 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109760365 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409420067192832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12670333632512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20758956 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1800946698973184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12670420105216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12530700534272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109750374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1804196053463706ü-œ'î%‹N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12511921673228 _runtime‚10092.753695206 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109750374 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1800946716012544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1813477757893120 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1813477399507456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12530700534272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20758953 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203700380 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25501373 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚975 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109760365 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409420067192832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12670333632512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20758956 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1800946698973184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12670420105216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12530700534272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109750374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1804196053463706 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203700380 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1800946698973184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396749628475392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚11849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203710373 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46260329 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1791684472340480 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7842829006278985e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396749647087616 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203710373 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409419962107904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93950008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1813477757893120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46260042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25501089 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12511581123226 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ïÿ‘î~4ï%/ _wandbruntime‚10092  _runtime‚10092ã¤ÊÈ ð%‡ 3loss/SRqqt1/assignment_loss‚0.39995313108607655 3loss/SRqqt2/assignment_loss‚0.44167883277734143 1loss/FBt1/assignment_loss‚0.03638385660049572 0loss/SRqqt2/detection_loss‚0.597958004536847 %loss/total_loss‚7.441228656134163 3loss/FBt2/assignment_loss‚8.176220058794226e-05 /loss/FRt2/detection_loss‚1.2243749554940704 1loss/SRqqt1/detection_loss‚0.8632982553840107 /loss/FBt1/detection_loss‚0.5265302728207427 trainer/global_step‚11849 % _timestamp‚1.7842829006294687e+09 _runtime‚10092.754802807 /loss/FBt2/detection_loss‚0.4996856602797793 _step‚976 /loss/FRt1/detection_loss‚0.5033570422267167 epoch‚29 0loss/FRt1/assignment_loss‚1.5924865789121407 0loss/FRt2/assignment_loss‚0.7554403038153529ºhé7ñ%Œ 3loss/FBt2/assignment_loss‚8.176220058794226e-05 /loss/FRt2/detection_loss‚1.2243749554940704 1loss/SRqqt1/detection_loss‚0.8632982553840107 /loss/FBt1/detection_loss‚0.5265302728207427 trainer/global_step‚11849 % _timestamp‚1.7842829006294687e+09 _runtime‚10092.754802807 /loss/FBt2/detection_loss‚0.4996856602797793 _step‚976 /loss/FRt1/detection_loss‚0.5033570422267167 epoch‚29 0loss/FRt1/assignment_loss‚1.5924865789121407 0loss/FRt2/assignment_loss‚0.7554403038153529 3loss/SRqqt1/assignment_loss‚0.39995313108607655 3loss/SRqqt2/assignment_loss‚0.44167883277734143 1loss/FBt1/assignment_loss‚0.03638385660049572 0loss/SRqqt2/detection_loss‚0.597958004536847 %loss/total_loss‚7.441228656134163ÐúÇÇò%‚(:’ –þçÒ²“Ï£ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïƒt³ÿó%‚(:ô –þçÒä´ó¤ disk.rbd3.out‚0 cpu‚26.027477989401508 disk.nvme0n1.in‚ 481.203125$ memory_percent‚97.32649326324463 disk.dm-1.out‚ 384.56640625 disk.sda.in‚0# proc.memory.rssMB‚ 2821.83984375* proc.memory.percent‚17.223143577575684 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 384.56640625( disk.nvme0c0n1.out‚47664.81982421875 disk.drbd2165.out‚0 disk.rbd0.in‚0 network.recv‚ 2.692357e+06# disk./.usageGB‚548.776294708252 disk.rbd1.in‚0.0625! disk.rbd1.out‚109.45751953125 disk.rbd3.in‚0 disk.dm-0.in‚ 481.203125 network.sent‚ 1.8797312e+07) disk./.usagePercent‚36.82883759185141 disk.dm-1.in‚0 disk.sda1.in‚0# disk.dm-0.out‚47664.81982421875( proc.memory.availableMB‚ 438.02734375& disk.nvme0n1.out‚47664.81982421875 disk.sda.out‚ 384.56640625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 481.203125 disk.rbd0.out‚0¼Áãô%‚(:’ ¥þçÒ…²¬¢ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡"¬Zõ%‚(:ì ¥þçÒ‰»‡¥ disk.sda3.out‚ 387.5078125( proc.memory.availableMB‚ 438.NÚ˽48046875# proc.memory.rssMB‚ 2821.83984375$ disk./.usageGB‚548.7773780822754 disk.sda.out‚ 387.5078125 disk.sda2.in‚0) disk./.usagePercent‚36.82891029799113 disk.nvme0n1.in‚ 481.203125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0! disk.rbd1.out‚109.52001953125 disk.rbd3.in‚0* proc.memory.percent‚17.223143577575684 disk.drbd2165.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 network.sent‚ 1.8860625e+07 disk.sda1.out‚0 disk.dm-1.out‚ 387.5078125 disk.dm-0.in‚ 481.203125 cpu‚26.027042719376993 proc.cpu.threads‚17! disk.dm-0.out‚47678.107421875 network.recv‚ 2.697738e+06$ disk.nvme0n1.out‚47678.107421875 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 481.203125 disk.dm-1.in‚0$ memory_percent‚97.32372760772705 disk.rbd0.in‚0& disk.nvme0c0n1.out‚47678.107421875²~`‘ö%‚(:† ´þçÒÉÏ¥¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›´/ ÷%‚(:õ ´þçÒŒæÞ¤ disk.sda2.out‚0 disk.rbd3.out‚0 cpu‚26.02656204214723 network.sent‚ 1.8865682e+07 disk.drbd2165.in‚0 disk.sda.out‚ 389.66015625 disk.sda3.out‚ 389.66015625 disk.dm-0.in‚ 481.91796875# proc.memory.rssMB‚ 2821.83984375* proc.memory.percent‚17.223143577575684 network.recv‚ 2.700931e+06* disk./.usagePercent‚36.828718292340454 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625$ disk.nvme0n1.out‚47694.990234375 disk.sda.in‚0$ memory_percent‚97.32544422149658$ disk./.usageGB‚548.7745170593262 disk.nvme0n1.in‚ 481.91796875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17! disk.dm-0.out‚47694.990234375! disk.rbd1.out‚109.52001953125( proc.memory.availableMB‚ 438.19921875 disk.dm-1.out‚ 389.66015625" disk.nvme0c0n1.in‚ 481.91796875& disk.nvme0c0n1.out‚47694.990234375Žke4ø%/ _wandbruntime‚10131  _runtime‚10131 Âo8Éù%à _runtime‚10131.982322755 _step‚977 & lr-AdamW/pg1‚0.0001727670195850076 & lr-AdamW/pg2‚0.0001727670195850076 trainer/global_step‚11899 % _timestamp‚1.7842829398559544e+09:]ÕÎú%È & lr-AdamW/pg2‚0.0001727670195850076 trainer/global_step‚11899 % _timestamp‚1.7842829398559544e+09 _runtime‚10131.982322755 _step‚977 & lr-AdamW/pg1‚0.0001727670195850076ÑzYà4û%/ _wandbruntime‚10131  _runtime‚101318†š0(ü%ûO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12580724748288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110205751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1811425039407870 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1820741892834304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94323976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1808161509425152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110195774 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1820742251212800 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1808161509425152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398387637643264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842829398567338e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20844458 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411108379563520 _step‚978 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94323974 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12720723308032 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1798863495262208 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1820742251212800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12580383409152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110195774 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25603081 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1820741892834304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94323976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46447808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12561186556412 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398387656255488 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1811424681818620 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1798863512301568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94323974 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204519748 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12580724748288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110205751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚411108284957184éxkM(ý%€P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12580724748288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110205751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1811425039407870 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1820741892834304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94323976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1808161509425152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110195774 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1820742251212800 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1808161509425152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398387637643264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842829398567338e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20844458 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411108379563520 _step‚978 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94323974 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12720723308032 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1798863495262208 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1820742251212800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12580383409152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110195774 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25603081 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1820741892834304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94323976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46447808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12561186556412 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398387656255488 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚94606336 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1811424681818620 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1798863512301568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94323974 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204519748 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12580724748288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110205751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚411108284957184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Ò~äiV4þ%/ _wandbruntime‚10132  _runtime‚10132ýì’˜K$ÿ%†N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94331608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94331608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110204826 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1799010022973440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12581737382400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12721664044032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46451371 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110214817 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 trainer/global_step‚11899 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25605420 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842829404690518e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46451626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1820890504876544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1799010005934080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204546425 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110204826 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204536432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25605168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚411142835530752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1811572203544214 _runtime‚10132.594802677 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12581396036096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batchµWÆÁ_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204546425 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12581737382400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚979 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411142730445824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398421066401792 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1820890146490880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438½bó5'€&‹N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94331608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94331608 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110204826 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1799010022973440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12581737382400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12721664044032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46451371 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110214817 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 trainer/global_step‚11899 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25605420 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842829404690518e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46451626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1820890504876544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1799010005934080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204546425 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110204826 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204536432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25605168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚411142835530752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1811572203544214 _runtime‚10132.594802677 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12581396036096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204546425 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12581737382400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚979 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411142730445824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398421066401792 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1820890146490880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438Ó*žî”4&/ _wandbruntime‚10132  _runtime‚10132îÿÙn‚&‰ &loss/total_loss‚7.4894450881848496 trainer/global_step‚11899 _runtime‚10132.596511212 % _timestamp‚1.7842829404711463e+09 _step‚980 3loss/FBt1/assignment_loss‚0.0011309237235418814 /loss/FRt1/detection_loss‚0.5150543301851446 1loss/SRqqt2/detection_loss‚0.6736953895634806 0loss/FRt1/assignment_loss‚1.4980358653026298 0loss/FRt2/assignment_loss‚0.8829255600953939 2loss/SRqqt1/assignment_loss‚0.4992317660650153 2loss/SRqqt2/assignment_loss‚0.5020700948423115 1loss/SRqqt1/detection_loss‚0.8481423113108343 /loss/FBt1/detection_loss‚0.2953259322551911 /loss/FBt2/detection_loss‚0.6030917871377564 epoch‚29 3loss/FBt2/assignment_loss‚0.0002460162463817385 /loss/FRt2/detection_loss‚1.1704951114571687r>­ƒ&Ž epoch‚29 3loss/FBt2/assignment_loss‚0.0002460162463817385 /loss/FRt2/detection_loss‚1.1704951114571687 &loss/total_loss‚7.4894450881848496 trainer/global_step‚11899 _runtime‚10132.596511212 % _timestamp‚1.7842829404711463e+09 _step‚980 3loss/FBt1/assignment_loss‚0.0011309237235418814 /loss/FRt1/detection_loss‚0.5150543301851446 1loss/SRqqt2/detection_loss‚0.6736953895634806 0loss/FRt1/assignment_loss‚1.4980358653026298 0loss/FRt2/assignment_loss‚0.8829255600953939 2loss/SRqqt1/assignment_loss‚0.4992317660650153 2loss/SRqqt2/assignment_loss‚0.5020700948423115 1loss/SRqqt1/detection_loss‚0.8481423113108343 /loss/FBt1/detection_loss‚0.2953259322551911 /loss/FBt2/detection_loss‚0.6030917871377564ÔÝÑYž„&‚(:“ ÃþçÒ‡Ñì¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬(•……&‚(:õ ÃþçÒõ²Ý¤ disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 482.74609375( disk.nvme0c0n1.out‚47702.49560546875$ memory_percent‚97.32556343078613( disk./.usagePercent‚36.8286371379521 disk.sda3.out‚ 391.0234375 disk.nvme0n1.in‚ 482.74609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 391.0234375# proc.memory.rssMB‚ 2821.83984375# disk./.usageGB‚548.773307800293# disk.dm-0.out‚47702.49560546875& disk.nvme0n1.out‚47702.49560546875 proc.cpu.threads‚17 network.recv‚ 2.704266e+06 disk.sda3.in‚0 disk.dm-1.out‚ 391.0234375 disk.dm-0.in‚ 482.74609375 disk.rbd0.out‚0 disk.rbd1.in‚0.0625 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚26.026105728094443 network.sent‚ 1.8870689e+07 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 438.1796875* proc.memory.percent‚17.223143577575684! disk.rbd1.out‚109.58251953125 disk.rbd3.in‚0 disk.sda1.out‚0˜ð–#†&‚(:’ ÒþçÒ–“ê£ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚254.229' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.84580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.¡ž*û‡&‚(:ð ÒþçÒŦ°¤ disk.drbd2165.out‚0 disk.rbd0.out‚0! disk.rbd1.out‚109.58251953125' proc.memory.availableMB‚ 438.1640625" disk.dm-0.out‚47712.1865234375 disk.nvme0n1.in‚ 483.4609375! disk.nvme0c0n1.in‚ 483.4609375' disk.nvme0c0n1.out‚47712.1865234375 disk.sda3.out‚ 391.2890625 disk.dm-1.in‚0 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2821.83984375) disk./.usagePercent‚36.82864097806511# disk./.usageGB‚548.773365020752 disk.sda1.out‚0 disk.sda.in‚0 cpu‚26.025652794600354 disk.drbd2165.in‚0 disk.sda2.in‚0 network.sent‚ 1.8933926e+07 network.recv‚ 2.709482e+06 disk.dm-0.in‚ 483.4609375 disk.sda2.out‚0$ memory_percent‚97.32565879821777* proc.memory.percent‚17.223143577575684 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚47712.1865234375 disk.sda.out‚ 391.2890625 disk.rbd3.out‚0 disk.dm-1.out‚ 391.2890625 disk.rbd3.in‚0ÂA{¸žˆ&‚(:“ áþçÒ‚ù½¢ gpu.0.gpu‚74.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ZÙ?‰&‚(:ö áþçÒմϤ& disk.nvme0n1.out‚47715.22509765625 disk.sda1.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 483.4609375 cpu‚26.025246527548862 disk.rbd0.out‚0 disk.rbd3.out‚0$ disk./.usageGB‚548.7733459472656 disk.rbd3.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0) disk./.usagePercent‚36.82863969802744 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 483.4609375 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 483.4609375# disk.dm-0.out‚47715.22509765625 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda.out‚ 391.55078125 disk.sda3.out‚ 391.55078125 disk.dm-1.in‚0$ memory_percent‚97.32580184936523& proc.memory.availableMB‚ 438.140625 disk.dm-1.out‚ 391.55078125! disk.rbd1.out‚109.70751953125 disk.sda2.in‚0# proc.memory.rssMB‚ 2821.83984375 proc.cpu.threads‚17 network.sent‚ 1.8938878e+07 network.recv‚ 2.713159e+06( disk.nvme0c0n1.out‚47715.22509765625* proc.memory.percent‚17.223143577575684^ŠD,4Š&/ _wandbruntime‚10171  _runtime‚10171Ôî×FÊ‹&Ä _step‚981 ' lr-AdamW/pg1‚0.00017094483484116302 ' lr-AdamW/pg2‚0.00017094483484116302 trainer/global_step‚11949 % _timestamp‚1.7842829797430432e+09 _runtime‚10171.86866283Ã$½%ÏŒ&É ' lr-AdamW/pg1‚0.00017094483484116302 ' lr-AdamW/pg2‚0.00017094483484116302 trainer/global_step‚11949 % _timestamp‚1.7842829797430432e+09 _runtime‚10171.86866283 _step‚981ÕdZo74&/ _wandbruntime‚10171  _runtime‚10171»ßß­(Ž&ûO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚982 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1828154639817728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12631420250112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1818801189488378 trainer/global_step‚11949 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1815523560906752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12631078910976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205365779 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1828154998196224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25707423 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412831161535488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110650226 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20931711 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1828154639817728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1818800831899128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1828154998196224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205355800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1815523577946112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12772166154240 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12612143593210 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508"ž‰&€P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25707423 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412831161535488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110650226 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20931711 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1828154639817728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1818800831899128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1828154998196224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205355800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1815523577946112 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12772166154240 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12612143593210 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚982 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1828154639817728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12631420250112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1818801189488378 trainer/global_step‚11949 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1815523560906752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12631078910976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205365779 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1828154998196224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268ÖF:és4&/ _wandbruntime‚10172  _runtime‚10172Æ̧… '‘&„N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412865724032512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400092522940416 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1806335556567040 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1828302893474304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1806335539527680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12612814097042 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1815670801936384 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1815670801936384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12632091537920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12613154647044 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1818948353624722 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12632432884224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20933456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20933453 _runtime‚10172.481265883 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12632091537920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205372484 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67598336 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110659278 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚11949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46642732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1828303251859968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94713208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1815670818975744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25709535 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205382477 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110669269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚400092504328192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110659278 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12632432884224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46642991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94713206 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25709279 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110669269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚400092504328192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110659278 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12632432884224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46642991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94713206 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25709279 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12613154647044 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1818948353624722 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12632432884224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20933456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20933453 _runtime‚10172.481265883 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12632091537920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205372484 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67598336 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110659278 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚11949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46642732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1828303251859968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94713208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1815670818975744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25709535 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205382477 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576ײ(Ûö4“&/ _wandbruntime‚10172  _runtime‚10172Њ¾Þ”&‚ 2loss/SRqqt2/assignment_loss‚0.4209566308544141 .loss/FBt1/detection_loss‚0.397882467051204 epoch‚29 _runtime‚10172.483889696 0loss/FRt1/assignment_loss‚1.5347643818304089 0loss/FBt1/assignment_loss‚0.0399160024535965 /loss/FRt1/detection_loss‚0.4970001563753093 /loss/FRt2/detection_loss‚1.2645756422470564 0loss/FBt2/detection_loss‚0.37860612298910773 &loss/total_loss‚7.2092815292257315 _step‚984 2loss/FBt2/assignment_loss‚0.035951408753715546 1loss/SRqqt2/detection_loss‚0.5708925183270159 trainer/global_step‚11949 0loss/FRt2/assignment_loss‚0.7733190967762762 2loss/SRqqt1/assignment_loss‚0.4132677186896383 0loss/SRqqt1/detection_loss‚0.882149382877988 # _timestamp‚1.78428298035855e+09“rà •&‡ 2loss/FBt2/assignment_loss‚0.035951408753715546 1loss/SRqqt2/detection_loss‚0.5708925183270159 trainer/global_step‚11949 0loss/FRt2/assignment_loss‚0.7733190967762762 2loss/SRqqt1/assignment_loss‚0.4132677186896383 0loss/SRqqt1/detection_loss‚0.882149382877988 # _timestamp‚1.78428298035855e+09 2loss/SRqqt2/assignment_loss‚0.4209566308544141 .loss/FBt1/detection_loss‚0.397882467051204 epoch‚29 _runtime‚10172.483889696 0loss/FRt1/assignment_loss‚1.5347643818304089 0loss/FBt1/assignment_loss‚0.0399160024535965 /loss/FRt1/detection_loss‚0.4970001563753093 /loss/FRt2/detection_loss‚1.2645756422470564 0loss/FBt2/detection_loss‚0.37860612298910773 &loss/total_loss‚7.2092815292257315 _step‚984Øûtr–&‚(:’ ðþçÒ¨Ž¤¤ gpu.0.gpu‚57.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚262.648' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.52960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ädyç—&‚(:Ü ðþçҤﶦ disk.drbd2165.in‚0! disk.rbd1.out‚109.70751953125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2821.83984375* proc.memory.percent‚17.223143577575684 disk.sda1.in‚0 disk.dm-0.out‚47723.25 disk.rbd3.out‚0 disk.nvme0n1.in‚ 484.2890625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 391.73046875 cpu‚26.02478454893956 network.sent‚ 1.8943892e+07 network.recv‚ 2.716379e+06 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.32725620269775 disk.nvme0c0n1.out‚47723.25! disk.nvme0c0n1.in‚ 484.2890625 disk.sda.in‚0 disk.dm-1.out‚ 391.73046875( proc.memory.availableMB‚ 437.90234375 disk.rbd0.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.82864993832881 disk.nvme0n1.out‚47723.25 disk.drbd2165.out‚0 disk.sda.out‚ 391.73046875 disk.dm-0.in‚ 484.2890625$ disk./.usageGB‚548.7734985351562kØV“˜&‚(:ˆ ÿþçһ擢 gpu.0.gpu‚58.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸OQ•ÿ™&‚(:ô ÿþçÒÙĉ¤# proc.memory.rssMB‚ 2821.83984375 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚47729.4951171875% proc.memory.availableMB‚ 437.96875* proc.memory.percent‚17.223143577575684 disk.rbd1.in‚0.0625! disk.rbd1.out‚109.77001953125$ disk./.usageGB‚548.7736473083496 disk.sda1.out‚0 disk.sda.out‚ 392.73046875 disk.sda3.out‚ 392.73046875 disk.rbd3.in‚0 disk.dm-0.in‚ 485.00390625 disk.drbd2165.in‚0 cpu‚26.02434588204375 network.recv‚ 2.721665e+06 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 485.00390625 disk.dm-1.out‚ 392.73046875 proc.cpu.threads‚17" disk.dm-0.out‚47729.4951171875 network.sent‚ 1.9006926e+07% disk.nvme0n1.out‚47729.4951171875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚36.82865992262264 disk.nvme0n1.in‚ 485.00390625 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚97.32685089111328Çñ¸i4š&/ _wandbruntime‚10211  _runtime‚10211Ìj¨»Ç›&Á _runtime‚10211.772907192 _step‚985 % lr-AdamW/pg1‚0.000169126849928077 % lr-AdamW/pg2‚0.000169126849928077 trainer/global_step‚11999 % _timestamp‚1.7842830196473458e+09Ø&¸Ìœ&Æ % lr-AdamW/pg1‚0.000169126849928077 % lr-AdamW/pg2‚0.000169126849928077 trainer/global_step‚11999 % _timestamp‚1.7842830196473458e+09 _runtime‚10211.772907192 _step‚985Ùý!“r4&/ _wandbruntime‚10211  _runtime‚10211âL_šl ž&ûO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12682115751936 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1835567386801152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12823457379328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206201831 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46829972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25811286 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12662419530228 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1835567745179648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12681774412800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMoni}öC•tor.on_train_batch_start/requested_bytes.all.allocated‚1826177339568886 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1835567386801152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12681774412800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95087176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111104678 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977„Ú(Ÿ&€P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21018961 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111114655 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12682115751936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111114655 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414554088550400 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1822885612388352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95087174 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1822885612388352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25811014 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1813514579488768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206191852 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95087174 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21018958 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12662760080118 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11999 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12682115751936 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1835567386801152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12823457379328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206201831 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46829972 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25811286 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12662419530228 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1835567745179648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12681774412800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1826177339568886 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1835567386801152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12681774412800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95087176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111104678 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1823032853417984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25813260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414588536776192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401763960866816 $ _timestamp‚1.784283020259734e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206218529 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95094808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46833706 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1835715998843392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12824489436672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111113730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21020706 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12663771133952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1813661090160640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚10212.38551669 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12683128386048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206218529 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414588431691264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401763942254592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95094806 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206218529 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95094808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46833706 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1835715998843392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12824489436672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111113730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21020706 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12663771133952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1813661090160640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚10212.38551669 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZGDeviceStatsMonitor.on_traiIò+ì n_batch_end/active_bytes.small_pool.allocated‚12683128386048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206218529 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414588431691264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401763942254592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95094806 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1823032853417984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25813260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414588536776192ÛÝPϘ4£&/ _wandbruntime‚10212  _runtime‚10212a + ¤&† trainer/global_step‚11999 2loss/SRqqt2/assignment_loss‚0.4990208561741421 /loss/FRt2/detection_loss‚1.2026022773475669 % _timestamp‚1.7842830202621105e+09 0loss/FRt2/assignment_loss‚0.8375416715483099 1loss/FBt2/assignment_loss‚0.12238196240515743 1loss/SRqqt1/detection_loss‚0.8894614335287025 /loss/FBt2/detection_loss‚0.5965244660694753 _runtime‚10212.387461772 0loss/FRt1/assignment_loss‚1.6467963340132608 /loss/FRt1/detection_loss‚0.5324573089954481 1loss/SRqqt2/detection_loss‚0.5836105644990579 _step‚988 epoch‚29 3loss/SRqqt1/assignment_loss‚0.48922542233087796 2loss/FBt1/assignment_loss‚0.015740704321582505 0loss/FBt1/detection_loss‚0.28563072772017906 $loss/total_loss‚7.70099372895376°à©©¥&‹ 1loss/SRqqt1/detection_loss‚0.8894614335287025 /loss/FBt2/detection_loss‚0.5965244660694753 _runtime‚10212.387461772 0loss/FRt1/assignment_loss‚1.6467963340132608 /loss/FRt1/detection_loss‚0.5324573089954481 1loss/SRqqt2/detection_loss‚0.5836105644990579 _step‚988 epoch‚29 3loss/SRqqt1/assignment_loss‚0.48922542233087796 2loss/FBt1/assignment_loss‚0.015740704321582505 0loss/FBt1/detection_loss‚0.28563072772017906 $loss/total_loss‚7.70099372895376 trainer/global_step‚11999 2loss/SRqqt2/assignment_loss‚0.4990208561741421 /loss/FRt2/detection_loss‚1.2026022773475669 % _timestamp‚1.7842830202621105e+09 0loss/FRt2/assignment_loss‚0.8375416715483099 1loss/FBt2/assignment_loss‚0.12238196240515743ÜܯäÝ’¦&‚(:‡ ŽÿçҜѥ¢ gpu.0.gpu‚31.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ԕȪ§&‚(:õ Žÿçҹ蠥 proc.cpu.threads‚17( disk./.usagePercent‚36.8289684117014 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 network.recv‚ 2.72514e+06 disk.dm-1.out‚ 393.1328125! disk.rbd1.out‚109.77001953125( proc.memory.availableMB‚ 437.55078125# proc.memory.rssMB‚ 2821.83984375$ disk./.usageGB‚548.7782440185547 disk.dm-0.in‚ 485.83203125& disk.nvme0n1.out‚47743.50048828125$ memory_percent‚97.32940196990967" disk.nvme0c0n1.in‚ 485.83203125 disk.rbd3.in‚0 cpu‚26.024840148184726 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚47743.50048828125* proc.memory.percent‚17.223143577575684 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 485.83203125 network.sent‚ 1.901206e+07( disk.nvme0c0n1.out‚47743.50048828125 disk.sda.out‚ 393.1328125 disk.sda3.in‚0 disk.sda3.out‚ 393.1328125jmÔ!4¨&/ _wandbruntime‚10229  _runtime‚10229÷Àž …=©&ÿz :"Purity/*FRt0SRqqt0FBt/event_purity‚0.31792932748794556 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6209113001823425 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2836189866065979 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6946169137954712 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3980759084224701 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24863073229789734 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5793851017951965 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7418745160102844 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6395003199577332 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.6166942119598389 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6580542922019958 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019104111939668655 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_5‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2846412658691406 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.634371280670166 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7211478352546692 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.273082971572876 particle/accuracy_3_of_3‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9015076160430908 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6553338766098022 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879648089408875 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3272780179977417 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7665334343910217 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_2_of_3‚0.5889363884925842 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8493581414222717 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48048391938209534 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20710189640522003 particle/accuracy_1_of_1‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37751907110214233 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7456538677215576 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6755146980285645 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3826476037502289 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877771735191345 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894083142280579 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24667999148368835 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6976416110992432 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6909176111221313 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9200964570045471 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19381223618984222 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7173753380775452 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583134651184 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4887423813343048 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7323186993598938 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3826024532318115 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6432855129241943 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2605341076850891 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 trainer/global_step‚11999 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6883378028869629 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.336103230714798 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5320833921432495 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.635246217250824 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.336509108543396 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21968095004558563 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6977630853652954 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8553307056427002 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2910196781158447 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6655293703079224 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05263783782720566 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05697768181562424 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02080521732568741 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7309901714324951 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.33646881580352783 *jet/accuracy_1_of_2‚0.8102056384086609 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5048030018806458 particle/accuracy_1_of_4‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6749810576438904 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.25332802534103394 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7421157956123352 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06327535212039948 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10962475091218948 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3721408545970917 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3272780179977417 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 (particle/accuracy‚0.8445793390274048 epoch‚29 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5693248510360718 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2064758539199829 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.32213541865348816 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2810956537723541 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4877427816390991 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02080521732568741 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12206155806779861 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 /particle/accuracy_4_of_4‚0.9509974122047424 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33661189675331116 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.797240674495697 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_3_of_4‚0.07695029675960541 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.020476918667554855 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.39995837211608887 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5526740550994873 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1508333384990692 % _timestamp‚1.7842830373127043e+09 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7192636728286743 _step‚989 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 )jet/accuracy_1_of_3‚0.932396411895752 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9979296326637268 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6473297476768494 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.747826874256134 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.676539421081543 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04324304684996605 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8596281409263611 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6420886516571045 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7663390040397644 +jet/accuracy_1_of_1‚0.36231446266174316 +particle/specificity‚0.9068993926048279 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977676868438721 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_5‚1 )loss/val_total_loss‚7.958721552617151 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3269891142845154 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6878899335861206 *jet/accuracy_1_of_4‚0.9804345369338989 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984768629074097 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02064754255115986 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7915688753128052 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963633418083191 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5689589977264404 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6480286717414856 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7708252668380737 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7776728868484497 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16598711907863617 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3408049941062927 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6490632891654968 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948004484176636 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2545819580554962 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.018890414386987686 particle/accuracy_2_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7328759431838989 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2950896620750427 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8553307056427002 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6661012768745422 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.29324033856391907 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6825647950172424 'particle/f_score‚0.6301857233047485 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957686066627502 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7239985466003418 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6432855129241943 +jet/accuracy_3_of_3‚0.06315599381923676 *jet/accuracy_2_of_4‚0.7920925617218018 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 _runtime‚10229.43913489 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918572306632996 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26152339577674866 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6610828042030334 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094˜O/ "ª&„{ 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984768629074097 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02064754255115986 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7915688753128052 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9963633418083191 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5689589977264404 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6480286717414856 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7708252668380737 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7776728868484497 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16598711907863617 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3408049941062927 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6490632891654968 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948004484176636 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2545819580554962 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.018890414386987686 particle/accuracy_2_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7328759431838989 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2950896620750427 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8553307056427002 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6661012768745422 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.29324033856391907 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6825647950172424 'particle/f_score‚0.6301857233047485 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957686066627502 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7239985466003418 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6432855129241943 +jet/accuracy_3_of_3‚0.06315599381923676 *jet/accuracy_2_of_4‚0.7920925617218018 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 _runtime‚10229.43913489 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918572306632996 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26152339577674866 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6610828042030334 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.31792932748794556 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6209113001823425 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2836189866065979 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6946169137954712 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3980759084224701 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24863073229789734 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5793851017951965 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7418745160102844 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6395003199577332 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.6166942119598389 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6580542922019958 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019104111939668655 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_5‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2846412658691406 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.634371280670166 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7211478352546692 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.273082971572876 particle/accuracy_3_of_3‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9015076160430908 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6553338766098022 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879648089408875 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3272780179977417 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7665334343910217 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_2_of_3‚0.5889363884925842 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8493581414222717 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48048391938209534 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20710189640522003 particle/accuracy_1_of_1‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37751907110214233 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969257116317749 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7456538677215576 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6755146980285645 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3826476037502289 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877771735191345 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894083142280579 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24667999148368835 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6976416110992432 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6909176111221313 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9200964570045471 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19381223618984222 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7173753380775452 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583134651184 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4887423813343048 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7323186993598938 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3826024532318115 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6432855129241943 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2605341076850891 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 trainer/global_step‚11999 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6883378028869629 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.336103230714798 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5320833921432495 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.635246217250824 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt0SRqqt*FBt/Fë’j‚pRt_purity‚0.336509108543396 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21968095004558563 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6977630853652954 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8553307056427002 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2910196781158447 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6655293703079224 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05263783782720566 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05697768181562424 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02080521732568741 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7309901714324951 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.33646881580352783 *jet/accuracy_1_of_2‚0.8102056384086609 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5048030018806458 particle/accuracy_1_of_4‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6749810576438904 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.25332802534103394 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7421157956123352 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06327535212039948 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10962475091218948 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3721408545970917 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3272780179977417 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 (particle/accuracy‚0.8445793390274048 epoch‚29 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5693248510360718 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2064758539199829 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.32213541865348816 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2810956537723541 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4877427816390991 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02080521732568741 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.12206155806779861 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 /particle/accuracy_4_of_4‚0.9509974122047424 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33661189675331116 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.797240674495697 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_3_of_4‚0.07695029675960541 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.020476918667554855 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.39995837211608887 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5526740550994873 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1508333384990692 % _timestamp‚1.7842830373127043e+09 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7192636728286743 _step‚989 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 )jet/accuracy_1_of_3‚0.932396411895752 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955822229385376 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9979296326637268 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6473297476768494 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.747826874256134 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.676539421081543 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04324304684996605 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8596281409263611 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6420886516571045 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7663390040397644 +jet/accuracy_1_of_1‚0.36231446266174316 +particle/specificity‚0.9068993926048279 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977676868438721 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_5‚1 )loss/val_total_loss‚7.958721552617151 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3269891142845154 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6878899335861206 *jet/accuracy_1_of_4‚0.9804345369338989 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301Ý—âLò‘«&‚(:† ÿçÒôÊУ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚286.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎZ¡ïù¬&‚(:î ÿçÒ¬ú€¥$ disk./.usageGB‚548.8364219665527 disk.sda3.in‚0# proc.memory.rssMB‚ 2821.83984375 network.recv‚ 2.730143e+06 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.out‚ 396.96484375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 396.96484375 disk.rbd3.out‚0 cpu‚26.02998735033085 proc.cpu.threads‚17 network.sent‚ 1.907504e+07( disk./.usagePercent‚36.8328727826061 disk.nvme0n1.in‚ 486.546875 disk.drbd2165.out‚0 disk.sda2.out‚0! disk.rbd1.out‚109.83251953125 disk.rbd3.in‚0' proc.memory.availableMB‚ 437.6640625 disk.nvme0c0n1.in‚ 486.546875' disk.nvme0c0n1.out‚47822.6083984375% disk.nvme0n1.out‚47822.6083984375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 486.546875" disk.dm-0.out‚47822.6083984375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 396.96484375 disk.rbd1.in‚0.0625$ memory_percent‚97.32871055603027* proc.memory.percent‚17.223143577575684^JQ}œ­&‚(:‘ ¬ÿçÒÙÓˆ¤ gpu.0.gpu‚43.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚281.23' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.24600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—cdÿ®&‚(:ô ¬ÿçÒíÞÞ¤# disk.dm-0.out‚47835.82470703125 disk.nvme0c0n1.in‚ 486.546875 disk.drbd2165.in‚0* proc.memory.percent‚17.223143577575684 disk.dm-1.out‚ 399.05859375$ disk./.usageGB‚548.7775077819824 disk.rbd1.in‚0.0625! disk.rbd1.out‚109.83251953125& disk.nvme0n1.out‚47835.82470703125# proc.memory.rssMB‚ 2821.83984375 network.sent‚ 1.9132401e+07 network.recv‚ 2.735005e+06( disk./.usagePercent‚36.8289190022473 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 437.12109375 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 399.05859375 disk.sda2.out‚0 disk.dm-0.in‚ 486.546875 disk.nvme0n1.in‚ 486.546875 proc.cpu.threads‚17 disk.rbd3.in‚0 cpu‚26.029580759790633 disk.sda3.out‚ 399.05859375 disk.dm-1.in‚0 disk.sda1.out‚0$ memory_percent‚97.33202457427979 disk.rbd3.out‚0( disk.nvme0c0n1.out‚47835.82470703125 disk.sda2.in‚0ÞûÜž¯&‚(:“ »ÿçÒÏ—¬¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Zéïö°&‚(:ë »ÿçÒÔ¨¶¤* disk./.usagePercent‚36.828815063188394 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0! disk.rbd1.out‚109.89501953125 disk.rbd3.out‚0# memory_percent‚97.3320722579956 disk.nvme0n1.in‚487.375 disk.rbd0.out‚0 network.recv‚ 2.738804e+06 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 401.03515625( proc.memory.availableMB‚ 437.11328125# proc.memory.rssMB‚ 2821.83984375 disk.sda.in‚0 network.sent‚ 1.9137401e+07 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚47848.26708984375& disk.nvme0n1.out‚47848.26708984375* proc.memory.percent‚17.223143577575684 disk.sda3.out‚ 401.03515625 disk.dm-1.out‚ 401.03515625 cpu‚26.02915110330939 disk.nvme0c0n1.in‚487.375( disk.nvme0c0n1.out‚47848.26708984375$ disk./.usageGB‚548.7759590148926 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-0.in‚487.375ÓùLÉ4±&/ _wandbruntime‚10268  _runtime‚10268D¤—Ȳ& & lr-AdamW/pg1‚0.0001673131816115233 & lr-AdamW/pg2‚0.0001673131816115233 trainer/global_step‚12049 $ _timestamp‚1.784283076657147e+09 _runtime‚10268.782778469 _step‚990Y™›ùͳ&Ç _runtime‚10268.782778469 _step‚990 & lr-AdamW/pg1‚0.0001673131816115233 & lr-AdamW/pg2‚0.0001673131816115233 trainer/global_step‚12049 $ _timestamp‚1.784283076657147e+09ÞŽÈk4´&/ _wandbruntime‚10268  _runtime‚10268õæ}â(µ&üO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95589440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47054814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21108728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1832177247559680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403417863522304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95589442 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416310787877376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1835479872588703 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1844927792018944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12750544459264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47055084 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1832177230520320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25946086 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207228871 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚12049 _runtime‚10268.784192649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1832177230520320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12731096292255 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416310680688128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1844927433640448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207218892 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1832177247559680 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1844927792018944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1822748759257088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12750203120128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111629452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1844927433640448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12750544459264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403417882134528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12892905742848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1822748776296448 % _timestamp‚1.7842830766584296e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12892817165824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95589442 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416310787877376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1835479872588703 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1844927792018944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12750544459264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47055084 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1832177230520320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25946086 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207228871 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚12049 _runtime‚10268.784192649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1832177230520320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12731096292255 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416310680688128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1844927433640448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207218892 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1832177247559680 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1844927792018944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1822748759257088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12750203120128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111629452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1844927433640448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12750544459264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403417882134528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12892905742848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1822748776296448 % _timestamp‚1.7842830766584296e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12892817165824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95589442 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12751215747072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1845075687297024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1832324471549952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47058885 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21110473 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21110476 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416345128003584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207245569 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403451292280832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12751215747072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12731766796087 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1822895269928960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚12049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111638504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1822895286968320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1832324471549952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25948409 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95597072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207235576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95597074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12893924292608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚10269.396305102 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1835627394314409 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842830772704647e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111638504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47058598 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207235576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1832324488589312 _step‚992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1835627036725047 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12732107346089 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111648495 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1845076045682688 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207245569 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111648495 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95597072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12893835722752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25948125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403451310893056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616§Î Œ'¹&‹N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416345128003584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207245569 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403451292280832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12751215747072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12731766796087 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1822895269928960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚12049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111638504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1822895286968320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1832324471549952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25948409 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95597072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207235576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95597074 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12893924292608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚10269.396305102 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1835627394314409 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842830772704647e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111638504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47058598 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207235576 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1832324488589312 _step‚992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1835627036725047 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12732107346089 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111648495 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1845076045682688 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207245569 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111648495 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95597072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12893835722752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25948125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403451310893056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12751557093376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95597074 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416345235185664 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12751215747072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1845075687297024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1832324471549952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47058885 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21110473 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21110476 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269àòÛáC4º&/ _wandbruntime‚10269  _runtime‚10269íëz »&… 0loss/FRt1/assignment_loss‚1.5009186309407219 3loss/FBt1/assignment_loss‚0.0033152343668500102 1loss/FBt2/assignment_loss‚0.00612870196965386 /loss/FRt1/detection_loss‚0.4914834745055697 1loss/SRqqt1/detection_loss‚0.8792844525974547 0loss/FBt2/detection_loss‚0.45178099568539226 _runtime‚10269.399117211 0loss/FRt2/assignment_loss‚0.7566274287696326 /loss/FBt1/detection_loss‚0.3889359396043064 $loss/total_loss‚7.07683180904651 trainer/global_step‚12049 $ _timestamp‚1.784283077273792e+09 _step‚993 3loss/SRqqt2/assignment_loss‚0.44056888039394126 /loss/FRt2/detection_loss‚1.1661714226778739 1loss/SRqqt1/assignment_loss‚0.330013827562856 1loss/SRqqt2/detection_loss‚0.6616028199722577 epoch‚30ƒGÙU¼&Š 1loss/SRqqt1/assignment_loss‚0.330013827562856 1loss/SRqqt2/detection_loss‚0.6616028199722577 epoch‚30 0loss/FRt1/assignment_loss‚1.5009186309407219 3loss/FBt1/assignment_loss‚0.0033152343668500102 1loss/FBt2/assignment_loss‚0.00612870196965386 /loss/FRt1/detection_loss‚0.4914834745055697 1loss/SRqqt1/detection_loss‚0.8792844525974547 0loss/FBt2/detection_loss‚0.45178099568539226 _runtime‚10269.399117211 0loss/FRt2/assignment_loss‚0.7566274287696326 /loss/FBt1/detection_loss‚0.3889359396043064 $loss/total_loss‚7.07683180904651 trainer/global_step‚12049 $ _timestamp‚1.784283077273792e+09 _step‚993 3loss/SRqqt2/assignment_loss‚0.44056888039394126 /loss/FRt2/detection_loss‚1.1661714226778739á)j¶´“½&‚(:ˆ ÊÿçÒÝ·ò¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚251.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.3626 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©<Þû¾&‚(:ð ÊÿçÒÀÆë¤ disk.rbd0.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚47861.09423828125( disk.nvme0c0n1.out‚47861.09423828125 disk.sda3.in‚0 disk.dm-1.out‚ 401.40625 disk.nvme0n1.in‚ 488.08984375 disk.drbd2165.in‚0 disk.dm-0.in‚ 488.08984375# disk.dm-0.out‚47861.09423828125# proc.memory.rssMB‚ 2821.83984375 network.sent‚ 1.9141544e+07) disk./.usagePercent‚36.82883707983634$ disk./.usageGB‚548.7762870788574 disk.drbd2165.out‚0 disk.sda.out‚ 401.40625 cpu‚26.028646270227583 disk.rbd0.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚109.89501953125 disk.rbd3.in‚0& proc.memory.availableMB‚ 437.484375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 488.08984375 disk.dm-1.in‚0$ memory_percent‚97.32980728149414* proc.memory.percent‚17.223143577575684 network.recv‚ 2.741608e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 401.40625òL$pž¿&‚(:“ ÙÿçҀ䑢 gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.574' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.714800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹H4-ÿÀ&‚(:ô ÙÿçÒê¢þ£ disk.sda.out‚ 401.67578125 disk.dm-1.out‚ 401.67578125 disk.rbd1.in‚0.0625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 488.91796875" proc.memory.availableMB‚437.75! disk.rbd1.out‚109.95751953125 disk.rbd3.out‚0 network.sent‚ 1.9204543e+07 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 cpu‚26.028222181082608 disk.sda2.in‚0 disk.sda3.out‚ 401.67578125 disk.dm-1.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚47869.62353515625 disk.rbd0.in‚0$ memory_percent‚97.32818603515625( disk./.usagePercent‚36.8288278635651$ disk./.usageGB‚548.7761497497559 disk.sda2.out‚0 disk.dm-0.in‚ 488.91796875 disk.nvme0n1.in‚ 488.91796875& disk.nvme0n1.out‚47869.62353515625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2821.83984375 disk.sda.in‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚47869.62353515625* proc.memory.percent‚17.223143577575684 network.recv‚ 2.746754e+06‰€IÄžÁ&‚(:“ èÿçÒŸÚö¢ gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚248.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.782199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´#ÚàÂ&‚(:õ èÿçÒ¥˜Š¥ disk.rbd3.in‚0 disk.sda1.in‚0 cpu‚26.02776491357597 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 489.6328125 disk.sda3.out‚ 401.9609375# disk.dm-0.out‚47878.43994140625 disk.dm-1.in‚0! disk.rbd1.out‚109.95751953125 network.sent‚ 1.9209589e+07 network.recv‚ 2.750159e+06* disk./.usagePercent‚36.828824279459624$ disk./.usageGB‚548.7760963439941 disk.sda.in‚0& disk.nvme0n1.out‚47878.43994140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚97.32882976531982( proc.memory.availableMB‚ 437.64453125 disk.sda3.in‚0 disk.dm-0.in‚ 489.6328125 disk.sda.out‚ 401.9609375! disk.nvme0c0n1.in‚ 489.6328125( disk.nvme0c0n1.out‚47878.43994140625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 401.9609375# proc.memory.rssMB‚ 2821.83984375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.223143577575684°`–Q4Ã&/ _wandbruntime‚10308  _runtime‚10308ö òðËÄ&Å ' lr-AdamW/pg1‚0.00016550394638002893 ' lr-AdamW/pg2‚0.00016550394638002893 trainer/global_step‚12099 % _timestamp‚1.7842831164315345e+09 _runtime‚10308.557132769 _step‚994 ßrÐÅ&Ê trainer/global_step‚12099 % _timestamp‚1.7842831164315345e+09 _runtime‚10308.557132769 _step‚994 ' lr-AdamW/pg1‚0.00016550394638002893 ' lr-AdamW/pg2‚0.00016550394638002893âíQ±4Æ&/ _wandbruntime‚10308  _runtime‚10308ÑVâŒÇ&ûO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95971042 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95971040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26049510 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1830074292850688 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1852340539002368 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1852340180623872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47245233 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12781372229273 _runtime‚10308.561366563 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112083904 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1852340539002368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1839539299041280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418033493023232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12944191574528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1839539282001920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95971040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21195981 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12800898621952 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1839539282001920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12800898621952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418033591823872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚995 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1842856022669211 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1839539299041280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1852340180623872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112093881 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112093881 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47245491 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1842855665079961 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208054944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416tãk(È&€P NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405089320060928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12801239961088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21195978 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1830074309890048 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12781712779163 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112093881 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112093881 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47245491 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1842855665079961 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208054944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95971042 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95971040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26049510 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1830074292850688 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1852340539002368 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1852340180623872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47245233 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12781372229273 _runtime‚10308.561366563 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112083904 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1852340539002368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1839539299041280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418033493023232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12944191574528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1839539282001920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95971040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21195981 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12800898621952 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1839539282001920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12800898621952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418033591823872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚995 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1842856022669211 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1839539299041280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1852340180623872 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12945305269248 _runtime‚10309.17153516 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12801911248896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208071628 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1839686540070912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418068054088704 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418067957392384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842831170453382e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47249319 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95978674 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26051593 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12945227185152 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21197723 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112092956 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1852488792666112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405122748819456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12782723832997 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95978672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208071628 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47249057 trainer/global_step‚12099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1839686523031552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1843003186805555 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1830220803522560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208081621 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1839686523031552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1839686540070912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208081621 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1852488792666112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95978674 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12801911248896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12782383282995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26051334 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚996 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112092956 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1852488434280448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12802252595200äð÷Ž$"Ë&‰N EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1852488792666112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95978674 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12801911248896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12782383282995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26051334 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚996 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112092956 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1852488434280448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12802252595200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12945305269248 _runtime‚10309.17153516 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12801911248896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208071628 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1839686540070912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418068054088704 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418067957392384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842831170453382e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47249319 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95978674 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26051593 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12945227185152 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21197723 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112092956 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1852488792666112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405122748819456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12782723832997 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95978672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208071628 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47249057 trainer/global_step‚12099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1839686523031552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1843003186805555 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1830220803522560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ ž²vë564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208081621 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1839686523031552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1839686540070912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208081621 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ä_€"E4Ì&/ _wandbruntime‚10309  _runtime‚10309I³ Í&‡ /loss/FRt1/assignment_loss‚1.531918650425256 3loss/FBt2/assignment_loss‚0.0002269628145262564 1loss/SRqqt1/detection_loss‚0.8627260113312475 trainer/global_step‚12099 % _timestamp‚1.7842831170489233e+09 _step‚997 epoch‚30 /loss/FRt2/assignment_loss‚0.730670595398316 0loss/FRt1/detection_loss‚0.49576185373935266 /loss/FRt2/detection_loss‚1.1732254445221235 /loss/FBt1/detection_loss‚0.3209113798773821 /loss/FBt2/detection_loss‚0.8115179323832746 _runtime‚10309.174315409 3loss/SRqqt2/assignment_loss‚0.46784644824552174 2loss/FBt1/assignment_loss‚0.004399548189868225 2loss/SRqqt1/assignment_loss‚0.4125786570941564 1loss/SRqqt2/detection_loss‚0.5628569247428448 %loss/total_loss‚7.374640408763869âêUÎ&Œ 2loss/SRqqt1/assignment_loss‚0.4125786570941564 1loss/SRqqt2/detection_loss‚0.5628569247428448 %loss/total_loss‚7.374640408763869 /loss/FRt1/assignment_loss‚1.531918650425256 3loss/FBt2/assignment_loss‚0.0002269628145262564 1loss/SRqqt1/detection_loss‚0.8627260113312475 trainer/global_step‚12099 % _timestamp‚1.7842831170489233e+09 _step‚997 epoch‚30 /loss/FRt2/assignment_loss‚0.730670595398316 0loss/FRt1/detection_loss‚0.49576185373935266 /loss/FRt2/detection_loss‚1.1732254445221235 /loss/FBt1/detection_loss‚0.3209113798773821 /loss/FBt2/detection_loss‚0.8115179323832746 _runtime‚10309.174315409 3loss/SRqqt2/assignment_loss‚0.46784644824552174 2loss/FBt1/assignment_loss‚0.004399548189868225å7\7“Ï&‚(:ˆ ÷ÿçÒ½¬Ö¢ gpu.0.gpu‚90.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅòoÝúÐ&‚(:ï ÷ÿçÒëÕ’¤ disk.sda1.in‚0 disk.rbd0.in‚0! disk.rbd1.out‚110.02001953125 network.sent‚ 1.9214805e+07 disk.nvme0n1.in‚ 489.6328125& disk.nvme0n1.out‚47883.98193359375 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 489.6328125 cpu‚26.02731964089391$ disk./.usageGB‚548.7747993469238 disk.dm-0.in‚ 489.6328125* disk./.usagePercent‚36.828737236897986 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 402.90625 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2821.83984375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 402.90625* proc.memory.percent‚17.223143577575684 proc.cpu.threads‚17 network.recv‚ 2.753584e+06( disk.nvme0c0n1.out‚47883.98193359375$ memory_percent‚97.33202457427979 disk.drbd2165.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 402.90625 disk.sda2.out‚0# disk.dm-0.out‚47883.98193359375( proc.memory.availableMB‚ 437.12109375wÕqžÑ&‚(:“ †€èÒðÞÇ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†#rÒ&‚(:ø †€èÒçÛŽ¥& disk.nvme0n1.out‚47910.10986328125! disk.rbd1.out‚110.02392578125$ memory_percent‚97.33035564422607 network.recv‚ 2.758425e+06# disk.dm-0.out‚47910.10986328125 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 490.4609375 disk.dm-1.in‚0 network.sent‚ 1.9278059e+07$ disk./.usageGB‚548.7790069580078 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚47910.10986328125 disk.sda.in‚0 disk.sda.out‚ 405.53515625 disk.rbd1.in‚0.0625 disk.rbd3.out‚0( proc.memory.availableMB‚ 437.39453125 cpu‚26.026865697059197) disk./.usagePercent‚36.82901961320825 disk.sda2.out‚0 disk.dm-1.out‚ 405.53515625# proc.memory.rssMB‚ 2821.83984375 disk.dm-0.in‚ 490.4609375 disk.nvme0n1.in‚ 490.4609375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 405.53515625 disk.rbd3.in‚0* proc.memory.percent‚17.223143577575684 proc.cpu.threads‚17 disk.drbd2165.in‚0O7¯|4Ó&/ _wandbruntime‚10348  _runtime‚10348”¦šÎËÔ&Å _runtime‚10348.698325407 _step‚998 ' lr-AdamW/pg1‚0.00016369926043739182 ' lr-AdamW/pg2‚0.00016369926043739182 trainer/global_step‚12149 % _timestamp‚1.7842831565712812e+09)øƒ>ÐÕ&Ê ' lr-AdamW/pg1‚0.00016369926043739182 ' lr-AdamW/pg2‚0.00016369926043739182 trainer/global_step‚12149 % _timestamp‚1.7842831565712812e+09 _runtime‚10348.698325407 _step‚998æˆ(4Ö&/ _wandbruntime‚10348  _runtime‚10348Cæ’Ñ(×&üO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12149 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚406760739375104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1837399843483648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1859753285985792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112548333 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12851935462912 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1846901333483520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406760757987328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419756354488832 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1837399826444288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419756249396736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1850232172749719 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1850231815160469 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47436454 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚999 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12995510021632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26153223 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208900975 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12995596501504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1859752927607296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96352640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21283228 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21283231 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1846901333483520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47436183 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12831988716181 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1846901350522880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112538356 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842831565720153e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1846901350522880 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12851594123776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208890996 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1859753285985792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12995510021632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26153223 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208900975 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12995596501504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1859752927607296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96352640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21283228 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21283231 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1846901333483520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47436183 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12831988716181 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1846901350522880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112538356 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842831565720153e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1846901350522880 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12851594123776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208890996 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1859753285985792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96360274 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1859901539649536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419790697617408 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1847048574513152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1859901539649536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12996618053632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47440306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47440036 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112557399 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12852606750720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112557399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1850379336886063 % _timestamp‚1.7842831571945386e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21284973 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚12149 _step‚1000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112547408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12832999769903 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12852606750720 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1837546337116160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1847048574513152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208917673 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21284976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406794168133632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1850379694475425 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208907680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208907680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_tr6•ç›)ain_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1837546354155520 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1847048591552512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96360272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96360272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12852948097024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12852948097024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26155063 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208917673 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21284976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406794168133632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1850379694475425 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208907680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208907680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1837546354155520 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1847048591552512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96360272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96360272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12852948097024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12852948097024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26155063 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96360274 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1859901539649536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419790697617408 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1847048574513152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1859901539649536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12996618053632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47440306 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47440036 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112557399 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12852606750720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112557399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1850379336886063 % _timestamp‚1.7842831571945386e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21284973 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚12149 _step‚1000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112547408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12832999769903 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12852606750720 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1837546337116160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360è ×h4Ü&/ _wandbruntime‚10349  _runtime‚10349u6F{ Ý&† % _timestamp‚1.7842831571961904e+09 _step‚1001 2loss/SRqqt1/assignment_loss‚0.4415766362053248 /loss/FBt1/detection_loss‚0.5923804953190484 2loss/SRqqt2/assignment_loss‚0.5242366480230317 3loss/FBt2/assignment_loss‚4.415881447528136e-05 /loss/FRt2/detection_loss‚1.2131079094074417 1loss/SRqqt1/detection_loss‚0.9259878975194361 epoch‚30 0loss/FRt1/assignment_loss‚1.5045022605626315 0loss/FRt2/assignment_loss‚0.7804068543594261 1loss/SRqqt2/detection_loss‚0.6288656134991536 %loss/total_loss‚7.792340157039534 trainer/global_step‚12149 _runtime‚10349.32151287 1loss/FBt1/assignment_loss‚0.01449684686726116 /loss/FRt1/detection_loss‚0.5029470436707353 /loss/FBt2/detection_loss‚0.6637877927915681l·õ.Þ&‹ 0loss/FRt1/assignment_loss‚1.5045022605626315 0loss/FRt2/assignment_loss‚0.7804068543594261 1loss/SRqqt2/detection_loss‚0.6288656134991536 %loss/total_loss‚7.792340157039534 trainer/global_step‚12149 _runtime‚10349.32151287 1loss/FBt1/assignment_loss‚0.01449684686726116 /loss/FRt1/detection_loss‚0.5029470436707353 /loss/FBt2/detection_loss‚0.6637877927915681 % _timestamp‚1.7842831571961904e+09 _step‚1001 2loss/SRqqt1/assignment_loss‚0.4415766362053248 /loss/FBt1/detection_loss‚0.5923804953190484 2loss/SRqqt2/assignment_loss‚0.5242366480230317 3loss/FBt2/assignment_loss‚4.415881447528136e-05 /loss/FRt2/detection_loss‚1.2131079094074417 1loss/SRqqt1/detection_loss‚0.9259878975194361 epoch‚30ézrj“ß&‚(:ˆ •€èÒج¬¢ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚255.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø[hà&‚(:ö •€èÒ«ñ ¥ disk.rbd0.out‚0 disk.sda.in‚0 network.sent‚ 1.9283059e+07 disk.dm-1.out‚ 408.48828125( disk.nvme0c0n1.out‚52255.04443359375( proc.memory.availableMB‚ 530.12109375! disk.rbd1.out‚110.14892578125# disk.dm-0.out‚52255.04443359375 disk.nvme0n1.in‚ 5064.484375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 408.48828125 disk.rbd3.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 5064.484375" proc.memory.rssMB‚ 2639.7265625) proc.memory.percent‚16.11161231994629 cpu‚26.02639336035337* disk./.usagePercent‚37.129006425748145 disk.drbd2165.in‚0 disk.sda3.in‚0 network.recv‚ 2.761853e+06 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 5064.671875& disk.nvme0n1.out‚52255.04443359375 disk.sda.out‚ 408.48828125 disk.sda2.in‚0$ disk./.usageGB‚553.2490272521973 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚96.76439762115479_¸7P“á&‚(:ˆ ¤€èÒª—Т gpu.0.gpu‚78.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚250.216' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.0432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùI“êâ&‚(:ß ¤€èÒñž‘¥ proc.cpu.threads‚17$ disk./.usageGB‚553.2459449768066 disk.dm-1.out‚410.75 disk.rbd1.in‚0.0625 disk.dm-0.out‚52511.33984375 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 13692.390625 disk.sda2.out‚0 disk.drbd2165.out‚0$ memory_percent‚96.76587581634521( proc.memory.availableMB‚ 529.87890625) proc.memory.percent‚16.11161231994629 disk.dm-0.in‚ 13692.578125 cpu‚26.025971668275446 network.sent‚ 1.9346121e+07! disk.rbd1.out‚110.14892578125 disk.sda3.out‚410.75) disk./.usagePercent‚37.12879957166048# disk.nvme0n1.out‚52511.33984375 disk.rbd0.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2639.7265625 network.recv‚ 2.766366e+06! disk.nvme0n1.in‚ 13692.3984375% disk.nvme0c0n1.out‚52511.33984375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚410.75 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0]š<‘ã&‚(:† ³€èÒ÷–•¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚280.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºMä&‚(:ö ³€èÒÁýÕ¥ disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 21281.28125$ memory_percent‚96.76454067230225( proc.memory.availableMB‚ 530.09765625" disk.dm-0.out‚52519.2548828125 disk.sda2.out‚0) proc.memory.percent‚16.11161231994629 network.recv‚ 2.769965e+06! disk.rbd1.out‚110.22705078125' disk.nvme0c0n1.out‚52519.2548828125* disk./.usagePercent‚37.128597325708434 disk.rbd3.in‚0% disk.nvme0n1.out‚52519.2548828125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 412.65234375 proc.cpu.threads‚17 network.sent‚ 1.9351128e+07 disk.rbd3.out‚0 disk.dm-0.in‚ 21281.3984375 disk.sda3.in‚0" proc.memory.rssMB‚ 2639.7265625 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 21281.28125 disk.sda1.in‚0 disk.sda3.out‚ 412.65234375$ disk./.usageGB‚553.2429313659668 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 412.65234375 disk.dm-1.in‚0 cpu‚26.025523882315866çìñÚ4å&/ _wandbruntime‚10388  _runtime‚10388Œ÷ÚfÊæ&Ä % _timestamp‚1.7842831962980201e+09 _runtime‚10388.423628495 _step‚1002 & lr-AdamW/pg1‚0.0001618992396952179 & lr-AdamW/pg2‚0.0001618992396952179 trainer/global_step‚12199V© ÑÏç&É trainer/global_step‚12199 % _timestamp‚1.7842831962980201e+09 _runtime‚10388.423628495 _step‚1002 & lr-AdamW/pg1‚0.0001618992396952179 & lr-AdamW/pg2‚0.0001618992396952179êg^”i4è&/ _wandbruntime‚10388  _runtime‚10388­ 9.(é&üO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784283196299078e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1867165674590720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1867166032969216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209737027 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1867166032969216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26256864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96734240 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1867165674590720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1003 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96734242 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113002785 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47627075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12902630964736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12902289625600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209727048 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1854263384965120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21370481 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408432177301504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1854263402004480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21370478 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96734242 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1854263384965120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421478939495424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚12199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1857608322830227 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13046854965248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1844725360037888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12882945752979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13046762193920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1854263402004480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112992808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26256597 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12902630964736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968\&·¹Wê&P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1854263384965120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421478939495424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚12199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1857608322830227 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13046854965248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1844725360037888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12882945752979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13046762193920 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1854263402004480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112992808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26256597 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12902630964736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784283196299078e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1867165674590720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1867166032969216 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209737027 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1867166032969216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26256864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96734240 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1867165674590720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1003 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96734242 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113002785 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47627075 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12902630964736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12902289625600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209727048 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1854263384965120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21370481 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408432177301504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1854263402004480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21370478 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96734242 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ë›ÿ{4ë&/ _wandbruntime‚10389  _runtime‚10389IOe 'ì&‡N E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47630855 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12883616256811 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚1004 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1867313928247296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12883956806813 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚12199 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421513401523712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408465606060032 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12903302252544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842831969095247e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209753725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113001860 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26258900 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209753725 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96741874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13047795463680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1854410625994752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21372226 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1857755844555933 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421513502414336 _runtime‚10389.035363883 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13047877742080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26258632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12903643598848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1854410625994752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚408465624672256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209743732 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1854410643034112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12903643598848 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96741874 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47630855 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12883616256811 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚1004 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1867313928247296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12883956806813 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚12199 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421513401523712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408465606060032 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12903302252544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842831969095247e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209753725 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113001860 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26258900 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209753725 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96741874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13047795463680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1854410625994752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21372226 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1857755844555933 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421513502414336 _runtime‚10389.035363883 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13047877742080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26258632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12903643598848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1854410625994752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚408465624672256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209743732 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1854410643034112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12903643598848 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96741874 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1874578779952640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113447260 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1852050893631488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚12249 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1874578779952640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113447260 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1874578421574144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113457237 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784283236176194e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423201669891584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1861625436446720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12953326466560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21457731 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12933562239887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26359946 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1852050910670848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97115840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12933221689997 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1861625453486080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13098139046400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1861625436446720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12952985127424 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1874578421574144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1861625453486080K¯q(ü&P PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97115840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12933221689997 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1861625453486080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13098139046400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1861625436446720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12952985127424 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1874578421574144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1861625453486080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13098054663680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12952985127424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21457728 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410103615227904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97115842 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12953326466560 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97115842 _runtime‚10428.301950074 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210563100 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1874578779952640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113447260 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1852050893631488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚12249 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1874578779952640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113447260 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1874578421574144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113457237 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 $ _timestamp‚1.784283236176194e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423201669891584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1861625436446720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12953326466560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21457731 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12933562239887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26359946 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1852050910670848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496ïàçM64ý&/ _wandbruntime‚10428  _runtime‚10428A¼hƒ 'þ&‡N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12954339100672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210589777 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13099085451264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1861772677476352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97123474 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚12249 _runtime‚10428.911700879 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1874727033616384 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1874726675230720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113456312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12934573293721 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47821490 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410137062598656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47821244 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚423236129437696 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1861772677476352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1861772694515712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12953997754368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21459473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210589777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210579784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13099167729664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842832367858808e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97123474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97123472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26361771 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113466303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1865131994636441 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12954339100672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210589777 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13099085451264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1861772677476352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97123474 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 trainer/global_step‚12249 _runtime‚10428.911700879 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1874727033616384 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1874726675230720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113456312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12934573293721 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47821490 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410137062598656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47821244 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚423236129437696 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1861772677476352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1861772694515712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12953997754368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21459473 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210589777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210579784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13099167729664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842832367858808e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97123474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97123472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26361771 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113466303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1865131994636441 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ðpIÕh4€'/ _wandbruntime‚10428  _runtime‚10428ÀiÊù '† epoch‚30 _runtime‚10428.914578902 _step‚1009 0loss/FRt1/assignment_loss‚1.4668656094298178 1loss/SRqqt1/detection_loss‚0.8612017569899701 0loss/FRt2/assignment_loss‚0.8096907241062276 2loss/SRqqt2/assignment_loss‚0.4739968509933252 3loss/FBt2/assignment_loss‚0.0006296103104530866 1loss/SRqqt2/detection_loss‚0.6696277127499636 trainer/global_step‚12249 $ _timestamp‚1.784283236789248e+09 1loss/FBt1/assignment_loss‚0.01918482355478135 .loss/FRt2/detection_loss‚1.203256743288548 0loss/FBt1/detection_loss‚0.23023705753762946 %loss/total_loss‚7.247452949882364 2loss/SRqqt1/assignment_loss‚0.3878985789757527 /loss/FRt1/detection_loss‚0.5018407184882135 /loss/FBt2/detection_loss‚0.6230227634576815OoŸŒ‚'‹ epoch‚30 _runtime‚10428.914578902 _step‚1009 0loss/FRt1/assignment_loss‚1.4668656094298178 1loss/SRqqt1/detection_loss‚0.8612017569899701 0loss/FRt2/assignment_loss‚0.8096907241062276 2loss/SRqqt2/assignment_loss‚0.4739968509933252 3loss/FBt2/assignment_loss‚0.0006296103104530866 1loss/SRqqt2/detection_loss‚0.6696277127499636 trainer/global_step‚12249 $ _timestamp‚1.784283236789248e+09 1loss/FBt1/assignment_loss‚0.01918482355478135 .loss/FRt2/detection_loss‚1.203256743288548 0loss/FBt1/detection_loss‚0.23023705753762946 %loss/total_loss‚7.247452949882364 2loss/SRqqt1/assignment_loss‚0.3878985789757527 /loss/FRt1/detection_loss‚0.5018407184882135 /loss/FBt2/detection_loss‚0.6230227634576815ñps€žƒ'‚(:“ ï€èÒô°Å¢ gpu.0.gpu‚90.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òs„'‚(:ö ï€èÒ¯ãÖ¤" disk.dm-0.out‚52567.3251953125 disk.sda3.in‚0 disk.dm-1.out‚ 415.8671875 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda.out‚ 415.8671875 disk.dm-1.in‚0% disk.nvme0n1.out‚52567.3251953125 disk.sda2.in‚0 disk.rbd0.out‚0# disk./.usageGB‚553.247444152832 disk.sda3.out‚ 415.8671875 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚16.11161231994629" disk.nvme0n1.in‚51136.00390625$ disk.nvme0c0n1.in‚51136.00390625 cpu‚26.02371398859939 proc.cpu.threads‚17 network.sent‚ 1.9429308e+07 disk.sda.in‚0# memory_percent‚96.7648983001709" proc.memory.rssMB‚ 2639.7265625 network.recv‚ 2.785125e+06* disk./.usagePercent‚37.128900182621436 disk.rbd3.out‚0 disk.dm-0.in‚51136.00390625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚52567.3251953125' proc.memory.availableMB‚ 530.0390625 disk.drbd2165.out‚0! disk.rbd1.out‚110.35205078125 disk.sda2.out‚0 disk.rbd1.in‚0.0625½D®Žž…'‚(:“ þ€èҸݚ¤ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š=×ó†'‚(:è þ€èÒè›Ò¥) disk./.usagePercent‚37.12889890258377 disk.sda3.in‚0" disk.nvme0n1.out‚ 52572.9921875 disk.drbd2165.out‚0 disk.rbd0.in‚0$ memory_percent‚96.76663875579834) proc.memory.percent‚16.11161231994629 disk.rbd1.in‚0.0625 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 58548.328125 disk.dm-0.out‚ 52572.9921875( proc.memory.availableMB‚ 529.75390625" proc.memory.rssMB‚ 2639.7265625 cpu‚26.02327041554391! disk.rbd1.out‚110.35205078125 disk.sda1.out‚0 disk.sda.out‚ 416.19921875 network.sent‚ 1.9494426e+07 network.recv‚ 2.789953e+06$ disk./.usageGB‚553.2474250793457 disk.dm-1.in‚0 disk.nvme0n1.in‚58548.5 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 416.19921875 disk.dm-0.in‚ 58548.5859375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.out‚ 416.19921875 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 52572.9921875 disk.sda2.out‚0 disk.rbd3.in‚0NÏ3K4‡'/ _wandbruntime‚10468  _runtime‚10468æûŸ'Æ _runtime‚10468.178331522 _step‚1010 ' lr-AdamW/pg1‚0.00015831365595307104 ' lr-AdamW/pg2‚0.00015831365595307104 trainer/global_step‚12299 % _timestamp‚1.7842832760519776e+09Õ‡Ûîщ'Ë trainer/global_step‚12299 % _timestamp‚1.7842832760519776e+09 _runtime‚10468.178331522 _step‚1010 ' lr-AdamW/pg1‚0.00015831365595307104 ' lr-AdamW/pg2‚0.00015831365595307104òÎ}Vø4Š'/ _wandbruntime‚10468  _runtime‚10468bkƒb(‹'üO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13149253025792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1868987487928320 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1872360265401993 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚10468.178580943 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1881991168557568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97497440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1868987504967680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1859376444264448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48008196 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113911689 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26463218 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411775071766528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21544978 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113901712 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1881991526936064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12983838176905 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12984178726795 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13003680629248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1868987487928320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13004021968384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1868987504967680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48008457 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21544981 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1859376427225088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97497440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26463476 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1881991526936064 _step‚1011 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211409131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113901712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13149341602816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113911689 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1881991168557568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97497440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1868987504967680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1859376444264448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48008196 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113911689 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26463218 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411775071766528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21544978 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113901712 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1881991526936064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12983838176905 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12984178726795 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13003680629248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1868987487928320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13004021968384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1868987504967680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48008457 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21544981 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1859376427225088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97497440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26463476 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_trainWdŽB _batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1881991526936064 _step‚1011 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211409131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113901712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13149341602816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113911689 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚10468.788798988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211425829 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411808481912832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12984849230627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1882139780599808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411808500525056 trainer/global_step‚12299 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1882139780599808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48011930 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26465207 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113920755 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97505072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97505074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1859522954936320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1869134728957952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚1012 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21546723 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424958862266880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7842832766629999e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97505072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13005034602496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211415836 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113910764 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113910764 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13150266880512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1869134745997312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13004693256192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1869134745997312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12985189780629 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424958748793344°’l''ŒN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13005034602496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1872507787127587 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1882139780599808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48011930 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26465207 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113920755 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97505072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97505074 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1859522954936320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1869134728957952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚1012 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21546723 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424958862266880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7842832766629999e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97505072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13005034602496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211415836 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113910764 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113910764 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13150266880512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1869134745997312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13004693256192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1869134745997312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12985189780629 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424958748793344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1882139422214144 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _runtime‚10468.788798988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211425829 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411808481912832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12984849230627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1882139780599808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411808500525056 trainer/global_step‚12299 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1889403915540992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426646913447424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1876349539409920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114366141 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13054717470208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114366141 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212235204 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413446491080704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 _runtime‚10508.442800106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21632228 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13054376131072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13034795213703 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426647024830976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1876349539409920 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114366141 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13054717470208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114366141 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212235204 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413446491080704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 _runtime‚10508.442800106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21632228 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13054376131072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13034795213703 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426647024830976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1889403915540992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426646913447424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114375207 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1876496797478912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26568579 _runtime‚10509.08490678 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 $ _timestamp‚1.784283316958991e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21633976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413479938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13055730104320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1889552527583232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212261881 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97886674 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1866848488529920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1879883937208095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1876496780439552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26568850 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212251888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97886672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114365216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13201538557952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1876496797478912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1889552527583232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114365216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13035465717535 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48202552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212261881 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1889552169197568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48202826 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21633973 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1879884294797457 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97886674 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212251888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114375207 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360_™Lœ¡'ŠN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48202552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212261881 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1889552169197568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48202826 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21633973 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1879884294797457 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97886674 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212251888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114375207 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426681371924480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114375207 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1876496797478912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26568579 _runtime‚10509.08490678 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 $ _timestamp‚1.7842833Úú®èt16958991e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21633976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413479938451456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13055730104320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1889552527583232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212261881 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97886674 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1866848488529920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1879883937208095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1876496780439552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26568850 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212251888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97886672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114365216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13201538557952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1876496797478912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1889552527583232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114365216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13035465717535ø3;b4¢'/ _wandbruntime‚10509  _runtime‚10509WÖv £'„ 2loss/SRqqt1/assignment_loss‚0.3616579073374817 2loss/SRqqt2/assignment_loss‚0.4581768930351209 /loss/FBt2/detection_loss‚0.5356165205680385 0loss/FRt1/assignment_loss‚1.5214662554416252 1loss/SRqqt1/detection_loss‚0.8657783690199241 .loss/FBt1/detection_loss‚0.412211320327365 %loss/total_loss‚7.244610450695442 trainer/global_step‚12349 _runtime‚10509.086365404 epoch‚30 _step‚1017 1loss/FBt1/assignment_loss‚0.00928766193385642 2loss/FBt2/assignment_loss‚0.009291519080930859 /loss/FRt1/detection_loss‚0.5055549472458494 /loss/FRt2/detection_loss‚1.1367366412535327 1loss/SRqqt2/detection_loss‚0.5986077911704791 $ _timestamp‚1.784283316960965e+09 0loss/FRt2/assignment_loss‚0.8302246242812382‹Û'̤'‰ 0loss/FRt2/assignment_loss‚0.8302246242812382 2loss/SRqqt1/assignment_loss‚0.3616579073374817 2loss/SRqqt2/assignment_loss‚0.4581768930351209 /loss/FBt2/detection_loss‚0.5356165205680385 0loss/FRt1/assignment_loss‚1.5214662554416252 1loss/SRqqt1/detection_loss‚0.8657783690199241 .loss/FBt1/detection_loss‚0.412211320327365 %loss/total_loss‚7.244610450695442 trainer/global_step‚12349 _runtime‚10509.086365404 epoch‚30 _step‚1017 1loss/FBt1/assignment_loss‚0.00928766193385642 2loss/FBt2/assignment_loss‚0.009291519080930859 /loss/FRt1/detection_loss‚0.5055549472458494 /loss/FRt2/detection_loss‚1.1367366412535327 1loss/SRqqt2/detection_loss‚0.5986077911704791 $ _timestamp‚1.784283316960965e+09ùžÙó¡¥'‚(:’ ºèÒ›ø£ gpu.0.gpu‚83.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l*çý¦'‚(:ò ºèÒ’š¥" disk.nvme0n1.in‚87798.33984375$ disk.nvme0n1.out‚52608.373046875 disk.sda2.in‚0 disk.sda3.out‚ 417.296875$ memory_percent‚96.74351215362549! proc.memory.rssMB‚ 2715.390625* proc.memory.percent‚16.573429107666016 disk.sda1.in‚0 network.recv‚ 2.805689e+06 disk.rbd3.in‚0( proc.memory.availableMB‚ 533.54296875& disk.nvme0c0n1.out‚52608.373046875 disk.sda.out‚ 417.296875 disk.dm-1.in‚0 disk.sda1.out‚0 network.sent‚ 1.9570876e+07) disk./.usagePercent‚37.12923324842349 disk.rbd1.in‚0.0625! disk.dm-0.out‚52608.373046875 disk.drbd2165.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 417.296875 disk.rbd3.out‚0 disk.dm-0.in‚87798.33984375 cpu‚26.02150796565306 disk.sda2.out‚0! disk.rbd1.out‚110.53955078125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚87798.33984375 proc.cpu.threads‚17 disk.sda3.in‚0$ disk./.usageGB‚553.2524070739746Üóâ§'‚(:’ ÉèÒ¡Ö¢ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èp†/¨'‚(:ö ÉèÒÆæÒ¤ disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0* proc.memory.percent‚16.622114181518555 network.sent‚ 1.9633879e+07 network.recv‚ 2.810745e+06 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 95409.9453125& disk.nvme0n1.out‚52612.88720703125( disk.nvme0c0n1.out‚52612.88720703125 disk.sda.out‚ 417.484375# disk.dm-0.out‚52612.88720703125" proc.memory.rssMB‚ 2723.3671875) disk./.usagePercent‚37.12923376043855 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 95409.9453125 disk.sda2.in‚0! disk.rbd1.out‚110.60205078125 proc.cpu.threads‚17# memory_percent‚96.7418909072876( proc.memory.availableMB‚ 533.80859375# disk.nvme0c0n1.in‚ 95409.9453125$ disk./.usageGB‚553.2524147033691 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 417.484375 disk.dm-1.out‚ 417.484375 disk.rbd3.in‚0 cpu‚26.021124769905708 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 ¤Ÿ‘©'‚(:† ØèÒ‚»È¢ gpu.0.gpu‚57.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÕƪ'‚(:û ØèÒÛÒ¢¥! disk.nvme0n1.in‚ 102600.171875 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚52622.43994140625 disk.sda.in‚0 network.sent‚ 1.9638934e+07 network.recv‚ 2.814406e+06 disk.dm-0.in‚102600.22265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.622114181518555) disk./.usagePercent‚37.12925219298102 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.dm-1.out‚ 417.6015625& disk.nvme0n1.out‚52622.43994140625 disk.sda.out‚ 417.6015625" proc.memory.rssMB‚ 2723.3671875 disk.sda3.out‚ 417.6015625 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚102600.0859375 disk.rbd3.in‚0# disk.dm-0.out‚52622.43994140625( proc.memory.availableMB‚ 533.30859375 cpu‚26.02066680689887 disk.sda2.in‚0 disk.dm-1.in‚0# memory_percent‚96.7449426651001 proc.cpu.threads‚17 disk.sda3.in‚0! disk.rbd1.out‚110.60205078125 disk.rbd0.in‚0$ disk./.usageGB‚553.2526893615723 disk.rbd3.out‚0‡ÁR4«'/ _wandbruntime‚10551  _runtime‚10551Æõhsˬ'Å ' lr-AdamW/pg1‚0.00015474811632017142 ' lr-AdamW/pg2‚0.00015474811632017142 trainer/global_step‚12399 $ _timestamp‚1.784283358986831e+09 _runtime‚10551.112465773 _step‚1018Ü׃sЭ'Ê ' lr-AdamW/pg1‚0.00015474811632017142 ' lr-AdamW/pg2‚0.00015474811632017142 trainer/global_step‚12399 $ _timestamp‚1.784283358986831e+09 _runtime‚10551.112465773 _step‚1018úÚð·4®'/ _wandbruntime‚10551  _runtime‚10551Õ^ó(¯'ýO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114810616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1874027511451648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1887112923152259 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213071256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21719481 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415117947619328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114810616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚12399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1883711607930880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98260642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26669355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1896817020902912 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1883711590891520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13105412972032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98260640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13105071632896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98260642 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1887112565563009 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1896816662524416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428369613972480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10551.113775476 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98260640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26669610 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415117929007104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13085071150721 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842833589879947e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21719478 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13085411700611 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1887112565563009 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1896816662524416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428369613972480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚10551.113775476 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98260640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26669610 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415117929007104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13085071150721 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842833589879947e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21719478 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13085411700611 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1883858848960512 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1896965274566656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428404063102976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1883858831921152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1874174005084160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48392571 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13252686725120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213087940 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213097933 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1020 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13106425606144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26671605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415151376377856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98268274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114819668 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13106425606144 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7842833596184893e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114829659 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13106084259840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13086082204443 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98268272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1883858848960512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114829659 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98268272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114819668 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1874174022123520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48392831 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169Áÿ ÷'³'ŒN TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13252600252416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1896965274566656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213087940 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7842833596184893e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114829659 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13106084259840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13086082204443 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98268272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1883858848960512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114829659 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98268272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114819668 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1874174022123520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48392831 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1883858848960512 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1896965274566656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428404063102976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1883858831921152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1874174005084160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48392571 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13252686725120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213087940 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213097933 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1020 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13106425606144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26671605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415151376377856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98268274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114819668 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13106425606144 B&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8379629254341125 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28436821699142456 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6961156129837036 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9866476655006409 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6634605526924133 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_3_of_3‚0.06325753778219223 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6067708730697632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6165692210197449 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1914091408252716 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05257266014814377 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6476811170578003 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46091246604919434 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5484147071838379 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5395378470420837 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7941962480545044 +jet/accuracy_1_of_1‚0.35030943155288696 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948433637619019 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3390372395515442 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7256240248680115 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2037072628736496 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 *jet/accuracy_1_of_4‚0.9764662384986877 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.921123206615448 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9951696991920471 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5592572093009949 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29416006803512573 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8733671307563782 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7725974321365356 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.276324987411499 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 )loss/val_total_loss‚7.927553893777436 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7325419187545776 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921507239341736 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_3_of_4‚0.07936452329158783 particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6719088554382324 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7200739979743958 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.716110110282898 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48887038230895996 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7401738166809082 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7660937309265137 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2846514582633972 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024075647816061974 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 +jet/accuracy_2_of_2‚0.16362497210502625 +particle/specificity‚0.9083222150802612 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6494941115379333 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049317747354507446 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09448002278804779 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148059844971 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962906837463379 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_3_of_5‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028665104880928993 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3197496235370636 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7347577214241028 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29216131567955017 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08361399918794632 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02608899585902691 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6387484073638916 (particle/accuracy‚0.8460828065872192 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7152831554412842 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955142140388489 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.64203280210495 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2490648776292801 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24611541628837585 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 _runtime‚10570.766646569 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7460505962371826 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8578335642814636 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.¿B›Œ987532913684845 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29380565881729126 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.794937252998352 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35270798206329346 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6979946494102478 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7460505962371826 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24739685654640198 *jet/accuracy_2_of_4‚0.8237531781196594 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6766444444656372 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6601618528366089 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 epoch‚30 /particle/accuracy_3_of_4‚0.9979296326637268 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29430854320526123 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27604061365127563 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7170698642730713 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9965580701828003 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7213719487190247 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22414502501487732 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8379629254341125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6779873967170715 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17714360356330872 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7893670201301575 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12416666746139526 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.736356794834137 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7930830121040344 *jet/accuracy_1_of_3‚0.9349809288978577 trainer/global_step‚12399 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18965552747249603 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4218614101409912 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8733671307563782 *jet/accuracy_2_of_3‚0.5888460874557495 particle/accuracy_2_of_4‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04282044619321823 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7250944972038269 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6991299390792847 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21581943333148956 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23957885801792145 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6494941115379333 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2644142806529999 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2846514582633972 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5241667032241821 /particle/accuracy_4_of_4‚0.9555904865264893 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.8081004023551941 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6414765119552612 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.023762527853250504 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04378636181354523 6validation_average_jet_accuracy‚0.3910142481327057/U|V¹=»'³{ particle/accuracy_1_of_4‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6719088554382324 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7200739979743958 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.716110110282898 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48887038230895996 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7401738166809082 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7660937309265137 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2846514582633972 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024075647816061974 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 +jet/accuracy_2_of_2‚0.16362497210502625 +particle/specificity‚0.9083222150802612 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6494941115379333 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049317747354507446 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09448002278804779 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148059844971 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962906837463379 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_3_of_5‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028665104880928993 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3197496235370636 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7347577214241028 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29216131567955017 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08361399918794632 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02608899585902691 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6387484073638916 (particle/accuracy‚0.8460828065872192 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7152831554412842 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9955142140388489 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.64203280210495 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2490648776292801 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24611541628837585 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 _runtime‚10570.766646569 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7460505962371826 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919900298118591 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8578335642814636 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987532913684845 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29380565881729126 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.794937252998352 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35270798206329346 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6979946494102478 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7460505962371826 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24739685654640198 *jet/accuracy_2_of_4‚0.8237531781196594 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6766444444656372 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6601618528366089 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 epoch‚30 /particle/accuracy_3_of_4‚0.9979296326637268 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29430854320526123 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27604061365127563 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7170698642730713 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9965580701828003 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7213719487190247 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22414502501487732 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8379629254341125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6779873967170715 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17714360356330872 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7893670201301575 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12416666746139526 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.736356794834137 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7930830121040344 *jet/accuracy_1_of_3‚0.9349809288978577 trainer/global_step‚12399 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18965552747249603 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4218614101409912 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8733671307563782 *jet/accuracy_2_of_3‚0.5888460874557495 particle/accuracy_2_of_4‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04282044619321823 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7250944972038269 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6991299390792847 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21581943333148956 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23957885801792145 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6494941115379333 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2644142806529999 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2846514582633972 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5241667032241821 /particle/accuracy_4_of_4‚0.9555904865264893 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.8081004023551941 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6414765119552612 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.023762527853250504 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04378636181354523 6validation_average_jet_accuracy‚0.3910142481327057 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5461871027946472 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27739059925079346 +particle/sensitivity‚0.6184731125831604 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8605906367301941 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4236111044883728 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028665104880928993 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6461820006370544 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6061449646949768 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5984223484992981 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 'particle/f_score‚0.6331044435501099 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.296875 _step‚1022 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9969227910041809 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24416762590408325 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938767552375793 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8912956714630127 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2591545581817627 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7324652671813965 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6130655407905579 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2332359254360199 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4467592239379883 jet/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025855910032987595 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2674673795700073 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879798889160156 $ _timestamp‚1.784283378640409e+09 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2577771544456482 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3869706094264984 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38592764735221863 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6807646751403809 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997653067111969 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.484938383102417 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5305044651031494 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7188421487808228 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900257587432861 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906298518180847 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.685109555721283 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8379629254341125 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28436821699142456 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6961156129837036 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9866476655006409 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6634605526924133 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_3_of_3‚0.06325753778219223 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6067708730697632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6165692210197449 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1914091408252716 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05257266014814377 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6476811170578003 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46091246604919434 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5484147071838379 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9240604639053345 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5395378470420837 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7941962480545044 +jet/accuracy_1_of_1‚0.35030943155288696 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948433637619019 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3390372395515442 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7256240248680115 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2037072628736496 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 *jet/accuracy_1_of_4‚0.9764662384986877 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.921123206615448 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9951696991920471 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5592572093009949 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29416006803512573 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8733671307563782 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7725974321365356 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.276324987411499 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 )loss/val_total_loss‚7.927553893777436 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7325419187545776 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921507239341736 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_3_of_4‚0.07936452329158783þþ}‘¼'‚(:† öèÒöÞô¢ gpu.0.gpu‚78.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­R’½'‚(:ü öèÒÜ÷ؤ disk.sda1.in‚0( disk.nvme0c0n1.out‚52638.43310546875 disk.sda3.out‚ 418.1171875 disk.dm-1.out‚ 418.1171875( proc.memory.availableMB‚ 441.56640625 disk.dm-0.in‚116382.0546875 disk.sda2.out‚0 disk.dm-1.in‚0) disk./.usagePercent‚37.12918639904472 disk.rbd3.in‚0 disk.sda2.in‚0# disk./.usageGB‚553.251708984375& disk.nvme0n1.out‚52638.43310546875 disk.drbd2165.in‚0 disk.rbd0.in‚0$ memory_percent‚97.30489253997803 network.sent‚ 1.9706798e+07 disk.rbd1.in‚0.0625 disk.rbd0.out‚0 network.recv‚ 2.822164e+06! disk.rbd1.out‚110.66455078125# disk.dm-0.out‚52638.43310546875 disk.sda.out‚ 418.1171875 disk.sda3.in‚0# proc.memory.rssMB‚ 2815.22265625$ disk.nvme0c0n1.in‚116382.0546875 cpu‚26.026966892720427 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0) proc.memory.percent‚17.18275547027588" disk.nvme0n1.in‚116382.0546875wԿ꓾'‚(:ˆ …‚èÒ–…±¢ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±vο'‚(:ú …‚èÒÞýï¤$ disk./.usageGB‚553.2487983703613% disk.nvme0c0n1.in‚124118.51171875 disk.sda2.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 441.73046875) proc.memory.percent‚17.18275547027588 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚124118.51171875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.976353e+07 disk.rbd1.in‚0.0625! disk.dm-0.out‚52647.154296875( disk./.usagePercent‚37.1289910652961 disk.sda.out‚ 420.28515625 disk.sda3.in‚0# disk.nvme0n1.in‚124118.51171875 disk.rbd0.in‚0# memory_percent‚97.3038911819458 disk.rbd0.out‚0 disk.sda3.out‚ 420.28515625$ disk.nvme0n1.out‚52647.154296875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 2.826613e+06& disk.nvme0c0n1.out‚52647.154296875! disk.rbd1.out‚110.72705078125 disk.rbd3.out‚0 cpu‚26.027468569487933 disk.dm-1.out‚ 420.28515625# proc.memory.rssMB‚ 2815.22265625 proc.cpu.threads‚17 disk.sda.in‚0-_øÞ’À'‚(:‡ ”‚èÒæ³â¢ gpu.0.gpu‚17.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Ê6ûÁ'‚(:ð ”‚èÒ’±õ¤ disk.drbd2165.out‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.30560779571533 disk.sda1.in‚0 disk.dm-0.in‚131488.31640625 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0! disk.rbd1.out‚110.72705078125 disk.drbd2165.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚52659.0947265625 disk.sda.out‚420.875) proc.memory.percent‚17.18275547027588 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚420.875 disk.dm-1.out‚420.875( proc.memory.availableMB‚ 441.44921875 network.sent‚ 1.9768642e+07 network.recv‚ 2.830137e+06) disk./.usagePercent‚37.12861908634884 disk.rbd0.in‚0% disk.nvme0c0n1.in‚131488.31640625 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2815.22265625 cpu‚26.02803996729647% disk.nvme0n1.out‚52659.0947265625$ disk./.usageGB‚553.2432556152344# disk.nvme0n1.in‚131488.31640625' disk.nvme0c0n1.out‚52659.0947265625»×g§4Â'/ _wandbruntime‚10610  _runtime‚10610œ—ÉÃ'à & lr-AdamW/pg1‚0.0001529731495076241 & lr-AdamW/pg2‚0.0001529731495076241 trainer/global_step‚12449 $ _timestamp‚1.784283418491769e+09 _runtime‚10610.617386981 _step‚1023Ô6NÎÄ'È & lr-AdamW/pg2‚0.0001529731495076241 trainer/global_step‚12449 $ _timestamp‚1.784283418491769e+09 _runtime‚10610.617386981 _step‚1023 & lr-AdamW/pg1‚0.0001529731495076241ÿL¼4Å'/ _wandbruntime‚10610  _runtime‚10610¤iìÚ¤Æ'ýO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115345367 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7842834184928262e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416805279033344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13153747912748 trainer/global_step‚12449 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1893003209023488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214108275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26804272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1896415456172076 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115345367 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98762906 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214098296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430126159477248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48613520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48613799 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚1024 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1893003209023488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1883261691219968 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1893003226062848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13173841679360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1883261708259328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1896415098582826 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115335390 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚10610.618619268 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98762908 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13173841679360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416805297645568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13173500340224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430126277152256 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190617706770[òž_ 42208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98762908 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1906176709363712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1893003226062848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21809251 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13320880443904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13320979506688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1906177067742208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98762906 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13173500340224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26804548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13153407362858 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21809248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1906176709363712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591Hë•1(Ç'‚P SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13173841679360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1883261708259328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1896415098582826 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115335390 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚10610.618619268 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98762908 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13173841679360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416805297645568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13173500340224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430126277152256 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1906177067742208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98762908 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1906176709363712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1893003226062848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21809251 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13320880443904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13320979506688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1906177067742208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98762906 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13173500340224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26804548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13153407362858 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21809248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1906176709363712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115345367 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7842834184928262e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416805279033344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13153747912748 trainer/global_step‚12449 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1893003209023488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214108275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26804272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1896415456172076 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115345367 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98762906 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214098296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430126159477248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48613520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48613799 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚1024 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1893003209023488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1883261691219968 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1893003226062848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1893150450053120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1893150450053120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48617663 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98770540 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214114980 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430160720491520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98770538 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13174512967168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416838707791872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214124973 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚10611.230389682 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1906325321405952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115344442 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13174854313472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48617375 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1896562620308420 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214114980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21810993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416838726404096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13154758966582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1896562977897782 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13174854313472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚12449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842834191045477e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13321994087424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13154418416580 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26806667 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13321907614720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98770538 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115344442 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21810996 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1883408201891840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48617375 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1896562620308420 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214114980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21810993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416838726404096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13154758966582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1896562977897782 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13174854313472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚12449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842834191045477e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13321994087424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13154418416580 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26806667 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13321907614720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98770538 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115344442 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21810996 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1883408201891840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1893150450053120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1893150450053120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48617663 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98770540 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214114980 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430160720491520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98770538 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13174512967168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416838707791872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214124973 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚10611.230389682 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1906325321405952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115344442 WIDeviceStatsMonitor.on_train_bat¯èM:Ých_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13174854313472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214934348 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1900365277544448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13224195842048 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1913589814725632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1913589814725632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48805100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115789842 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1900365260505088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21896501 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21896498 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1028 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431848916832768 trainer/global_step‚12499 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115799819 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1900365277544448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99144508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1900365260505088 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1903791248663334 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842834587398293e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13372086392320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13204023849766 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13224195842048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26908353 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1890587241852928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13204364399656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431848803352064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1913589456347136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13224537181184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13224537181184 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1890587224813568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99144508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99144506 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13372181260800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115799819 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418476716959744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214944327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48804851 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _runtime‚10650.865623852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26908599 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418476735571968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1913589456347136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚948684804Ô(Ù'‚P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1913589814725632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48805100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115789842 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1900365260505088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21896501 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21896498 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1028 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431848916832768 trainer/global_step‚12499 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115799819 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1900365277544448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99144508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1900365260505088 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1903791248663334 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842834587398293e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13372086392320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13204023849766 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13224195842048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26908353 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1890587241852928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13204364399656 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431848803352064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1913589456347136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13224537181184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13224537181184 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1890587224813568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99144508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99144506 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13372181260800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115799819 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418476716959744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214944327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48804851 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _runtime‚10650.865623852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26908599 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418476735571968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1913589456347136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214934348 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1900365277544448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13224195842048 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1913589814725632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0„‘°Á…4Ú'/ _wandbruntime‚10651  _runtime‚10651F¿â3Û'‡N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214951032 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21898243 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13205034903488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26910392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1900512518574080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1903939127978290 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842834593523068e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115798894 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115808885 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13373198647296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99152138 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1913738068389376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99152140 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214961025 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13225549815296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13225208468992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13373107980288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21898246 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_traM\ÅwÚin_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1900512518574080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214961025 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431883253698560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10651.478098044 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1913738068389376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418510164330496 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13205375453490 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚1029 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48808635 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115808885 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1900512501534720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431883362977792 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418510145718272 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1890733735485440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13225549815296 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431883362977792 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418510145718272 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1890733735485440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13225549815296 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21898243 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13205034903488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26910392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1900512518574080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1903939127978290 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842834593523068e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115798894 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115808885 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13373198647296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99152138 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1913738068389376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99152140 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214961025 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13225549815296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13225208468992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13373107980288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21898246 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1900512518574080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214961025 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431883253698560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10651.478098044 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1913738068389376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418510164330496 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13205375453490 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _step‚1029 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48808635 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115808885 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1900512501534720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116254271 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1907727329026048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚420148173498368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99526106 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1921002203330560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21983748 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27013479 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1907727311986688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚420148154886144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1911167398743842 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13254640336674 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7842834992068257e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1907727329026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116244294 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13254980886564 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99526108 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1921002561709056 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1907727311986688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116244294 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21983751 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27013209 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13423271072768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1921002561709056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433571425958912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976S˜É é'‚P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13275232683008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433571528953856 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99526106 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1921002203330560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21983748 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27013479 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1907727311986688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚420148154886144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.o‹q¼Êðn_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1911167398743842 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13254640336674 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7842834992068257e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1907727329026048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116244294 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13254980886564 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99526108 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1921002561709056 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1907727311986688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116244294 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21983751 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27013209 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13423271072768 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1921002561709056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433571425958912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1897912758407168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116254271 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1907727329026048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚420148173498368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49000813 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1907874553016320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 trainer/global_step‚12549 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1921150815372800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27015580 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215797077 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1907874570055680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13255991940398 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13255651390396 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116253346 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433605984142848 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13276245317120 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1921150815372800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1898059269079040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1033 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99533738 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49001076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13424301704704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 % _timestamp‚1.7842834998196244e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215787084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚420181602256896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99533738 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13275903970816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13275903970816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27015320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99533740 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116253346 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433605984142848 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13276245317120 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1921150815372800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1898059269079040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1033 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99533738 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49001076 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13424301704704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 % _timestamp‚1.7842834998196244e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215787084 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚420181602256896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99533738 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13275903970816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13275903970816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27015320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99533740 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49000813 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1907874553016320 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 trainer/global_step‚12549 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1921150815372800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27015580 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215797077 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1907874570055680‰ª˜¡é4í'/ _wandbruntime‚10691  _runtime‚10691ì¯UÜî'ˆ 0loss/FBt1/detection_loss‚0.39737886791958266 epoch‚31 _step‚1034 2loss/SRqqt1/assignment_loss‚0.3794020427961061 /loss/FRt2/detection_loss‚1.2192383364521326 1loss/SRqqt1/detection_loss‚0.9299075062348678 /loss/FBt2/detection_loss‚0.6781284940062147 &loss/total_loss‚7.4590287478875315 $ _timestamp‚1.784283499822522e+09 0loss/FRt1/assignment_loss‚1.5631702673648562 2loss/FBt1/assignment_loss‚0.014381544493096287 2loss/FBt2/assignment_loss‚0.009691660874274317 0loss/FRt1/detection_loss‚0.45852544049995014 _runtime‚10691.948189926 0loss/FRt2/assignment_loss‚0.7660126427915644 2loss/SRqqt2/assignment_loss‚0.4418431073285727 0loss/SRqqt2/detection_loss‚0.601348837126313 trainer/global_step‚12549 -Ÿ+ï' trainer/global_step‚12549 0loss/FBt1/detection_loss‚0.39737886791958266 epoch‚31 _step‚1034 2loss/SRqqt1/assignment_loss‚0.3794020427961061 /loss/FRt2/detection_loss‚1.2192383364521326 1loss/SRqqt1/detection_loss‚0.9299075062348678 /loss/FBt2/detection_loss‚0.6781284940062147 &loss/total_loss‚7.4590287478875315 $ _timestamp‚1.784283499822522e+09 0loss/FRt1/assignment_loss‚1.5631702673648562 2loss/FBt1/assignment_loss‚0.014381544493096287 2loss/FBt2/assignment_loss‚0.009691660874274317 0loss/FRt1/detection_loss‚0.45852544049995014 _runtime‚10691.948189926 0loss/FRt2/assignment_loss‚0.7660126427915644 2loss/SRqqt2/assignment_loss‚0.4418431073285727 0loss/SRqqt2/detection_loss‚0.601348837126313ŠØvkƒ’ð'‚(:‡ î‚èÒÁí‹¥ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚231.105' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.221 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñå…ÿñ'‚(:ô î‚èÒ—¸¦ disk.sda2.in‚0! disk.rbd1.out‚110.91455078125 disk.sda1.out‚0 disk.rbd0.in‚0# memory_percent‚97.3048210144043& proc.memory.availableMB‚ 441.578125 network.recv‚ 2.853576e+06 disk.dm-1.out‚ 426.17578125 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 144820.78125% disk.nvme0n1.out‚52711.7197265625' disk.nvme0c0n1.out‚52711.7197265625) proc.memory.percent‚17.18275547027588 cpu‚26.030893101109122 network.sent‚ 1.9928083e+07 disk.sda.in‚0 disk.sda3.out‚ 426.17578125 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2815.22265625 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 144820.78125) disk./.usagePercent‚36.82844103618087 disk.dm-1.in‚0 disk.nvme0n1.in‚ 144820.78125$ disk./.usageGB‚548.7703857421875 disk.sda.out‚ 426.17578125" disk.dm-0.out‚52711.7197265625 disk.drbd2165.out‚0 proc.cpu.threads‚17 Ô3“ò'‚(:ˆ ý‚èÒ™œÆ¢ gpu.0.gpu‚54.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡|)þó'‚(:ó ý‚èÒ£‰â¤ disk.sda2.out‚0 disk.sda3.out‚ 426.5546875 disk.sda1.in‚0 disk.dm-1.out‚ 426.5546875 disk.rbd3.in‚0$ memory_percent‚97.30510711669922 network.recv‚ 2.858936e+06) disk./.usagePercent‚36.82887061682332! disk.nvme0n1.in‚ 144821.609375 disk.sda3.in‚0) proc.memory.percent‚17.18275547027588# disk.nvme0c0n1.in‚ 144821.609375 disk.dm-1.in‚0! disk.rbd1.out‚110.97705078125$ disk./.usageGB‚548.7767868041992 disk.drbd2165.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚52719.1806640625# proc.memory.rssMB‚ 2815.22265625 network.sent‚ 1.9978256e+07% disk.nvme0n1.out‚52719.1806640625 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 144821.609375% proc.memory.availableMB‚ 441.53125' disk.nvme0c0n1.out‚52719.1806640625 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda2.in‚0 cpu‚26.03139303460816 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.out‚ 426.5546875° HIôô'‚(:’ ŒƒèÒÞñç¢ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚236.479' gpu.0.enforcedPowerLimitWatts‚50¤Ö‚©0.0( gpu.0.powerPercent‚47.29580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ÷„ õ'‚(:€ ŒƒèҚ˓¥ disk.rbd0.out‚0 disk.sda.out‚ 426.7265625 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚52729.07470703125 disk.dm-1.in‚0 network.sent‚ 1.9983267e+07 network.recv‚ 2.862211e+06! disk.rbd1.out‚110.97705078125* disk./.usagePercent‚36.828885721267845$ disk./.usageGB‚548.7770118713379& disk.nvme0n1.out‚52729.07470703125( proc.memory.availableMB‚ 441.52734375% disk.nvme0c0n1.in‚144822.32421875 disk.sda3.out‚ 426.7265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 426.7265625 disk.rbd1.in‚0.0625) proc.memory.percent‚17.18275547027588 cpu‚26.03185800425852 proc.cpu.threads‚17 disk.dm-0.in‚144822.32421875 disk.sda2.in‚0 disk.rbd3.in‚0$ memory_percent‚97.30513095855713# proc.memory.rssMB‚ 2815.22265625# disk.dm-0.out‚52729.07470703125 disk.rbd0.in‚0 disk.sda3.in‚0# disk.nvme0n1.in‚144822.32421875 disk.sda1.out‚0‘hi4ö'/ _wandbruntime‚10731  _runtime‚10731WÁÊ÷'Ä & lr-AdamW/pg2‚0.0001476808280656116 trainer/global_step‚12599 % _timestamp‚1.7842835396757472e+09 _runtime‚10731.801358965 _step‚1035 & lr-AdamW/pg1‚0.0001476808280656116™“nTÏø'É & lr-AdamW/pg1‚0.0001476808280656116 & lr-AdamW/pg2‚0.0001476808280656116 trainer/global_step‚12599 % _timestamp‚1.7842835396757472e+09 _runtime‚10731.801358965 _step‚1035‹vÜ•4ù'/ _wandbruntime‚10731  _runtime‚10731tí·É(ú'ýO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1928414950313984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1915089380507648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99907706 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13325928184832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13325928184832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13305597373472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1928415308692480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1918543548824350 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49189047 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13325586845696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1928414950313984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27118046 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22071001 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1928415308692480 _step‚1036 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216606452 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99907706 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116708723 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99907708 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435294005885440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49188778 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435294117268992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13474505844224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116698746 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421819592812544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1918543906413600 _runtime‚10731.802569581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22070998 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚12599 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99907708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1905238309040128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421819611424768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13305256823582 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1915089363468288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116708723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13474413072896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1915089380507648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476ÅoÏ\(û'‚P ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13305256823582 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1915089363468288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116708723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13474413072896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1915089380507648 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1928414950313984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1915089380507648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99907706 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13325928184832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13325928184832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13305597373472 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1928415308692480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1918543548824350 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49189047 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13325586845696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1928414950313984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27118046 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22071001 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1928415308692480 _step‚1036 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216606452 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99907706 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116708723 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99907708 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435294005885440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49188778 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435294117268992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13474505844224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116698746 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421819592812544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1918543906413600 _runtime‚10731.802569581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22070998 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚12599 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99907708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1905238309040128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421819611424768Œr>1#4ü'/ _wandbruntime‚10732  _runtime‚107320¦Ç 'ý'‡N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13326599472640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1918691428139306 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1905384819712000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27119874 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1915236604497920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22072743 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435328562979840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116717789 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13326940818944 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1928563562356224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216633129 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116707798 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1918691070549944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13306608427306 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚12599 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1915236621537280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99915338 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13475432129536 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49192617 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27120158 _runtime‚10732.414450124 _step‚1037 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49192904 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216623136 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216633129 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1918691428139306 D>D‘_)í`%eviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1905384819712000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27119874 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1915236604497920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22072743 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435328562979840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116717789 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13326940818944 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1928563562356224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216633129 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116707798 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1918691070549944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13306608427306 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚12599 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1915236621537280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99915338 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13475432129536 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49192617 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27120158 _runtime‚10732.414450124 _step‚1037 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49192904 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216623136 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216633129 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QS(Œ(P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100289308 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117153198 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49379954 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1935827697297408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7842835799196734e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100289306 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1922451414949888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1912563825594368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1935827697297408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13525617471488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27221983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22158248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437016559633408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117153198 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423491049351168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13376623686656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423491030738944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1935828055675904 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1922451414949888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1935828055675904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117163175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1925919698904858 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100289306 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13356213860380 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117153198 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423491049351168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13376623686656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423491030738944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1935828055675904 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1922451414949888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1935828055675904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117163175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1925919698904858 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100289306 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13356213860380 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100289308 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117153198 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49379954 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1935827697297408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7842835799196734e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100289306 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1922451414949888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1912563825594368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1935827697297408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13525617471488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27221983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22158248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437016559633408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541«çí4Ž(/ _wandbruntime‚10772  _runtime‚10772×Ãó>'(‹N _runtime‚10772.657280536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13357224914214 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22159996 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1922598655979520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217469181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100296938 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1912710353305600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49383711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217459188 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423524478109696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217459188 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117172241 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437051114474496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169÷ŒiÔ'(N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217459188 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117172241 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437051114474496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚10772.657280536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13357224914214 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22159996 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1922598655979520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217469181 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100296938 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1912710353305600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49383711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217459188 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423524478109696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0‘·Ez4‘(/ _wandbruntime‚10772  _runtime‚10772Jó’(‰ 0loss/FRt1/assignment_loss‚1.4902198750903641 1loss/SRqqt1/detection_loss‚0.8946945686572386 1loss/SRqqt2/detection_loss‚0.6190021202206911 0loss/FBt2/detection_loss‚0.33680560534412507 epoch‚31 $ _timestamp‚1.784283580534639e+09 /loss/FBt1/detection_loss‚0.5210251716149702 %loss/total_loss‚7.164566712506308 _step‚1042 0loss/FRt2/assignment_loss‚0.8115968999019169 2loss/SRqqt2/assignment_loss‚0.4425262496652524 2loss/FBt1/assignment_loss‚0.001017667955707176 0loss/FRt1/detection_loss‚0.49900569915456044 2loss/SRqqt1/assignment_loss‚0.3751311770513041 3loss/FBt2/assignment_loss‚0.0013901653335579578 /loss/FRt2/detection_loss‚1.1721515125166198 trainer/global_step‚12649 _runtime‚10772.659992426„Ó3G“(Ž _runtime‚10772.659992426 0loss/FRt1/assignment_loss‚1.4902198750903641 1loss/SRqqt1/detection_loss‚0.8946945686572386 1loss/SRqqt2/detection_loss‚0.6190021202206911 0loss/FBt2/detection_loss‚0.33680560534412507 epoch‚31 $ _timestamp‚1.784283580534639e+09 /loss/FBt1/detection_loss‚0.5210251716149702 %loss/total_loss‚7.164566712506308 _step‚1042 0loss/FRt2/assignment_loss‚0.8115968999019169 2loss/SRqqt2/assignment_loss‚0.4425262496652524 2loss/FBt1/assignment_loss‚0.001017667955707176 0loss/FRt1/detection_loss‚0.49900569915456044 2loss/SRqqt1/assignment_loss‚0.3751311770513041 3loss/FBt2/assignment_loss‚0.0013901653335579578 /loss/FRt2/detection_loss‚1.1721515125166198 trainer/global_step‚12649’6tâ<“”(‚(:ˆ ȃèÒžñŠ£ gpu.0.gpu‚78.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùPÅ•(‚(:ù ȃèÒ…ì¤$ disk.nvme0c0n1.in‚144824.6953125' proc.memory.availableMB‚ 441.2265625 disk.sda1.in‚0 disk.sda.out‚ 428.8359375 disk.rbd1.in‚0.0625 disk.rbd3.in‚0$ disk./.usageGB‚548.7770729064941% disk.nvme0n1.out‚52757.1982421875 disk.sda1.out‚0 disk.sda2.out‚0! disk.rbd1.out‚111.10595703125# proc.memory.rssMB‚ 2815.22265625 network.recv‚ 2.884744e+06" disk.nvme0n1.in‚144824.6953125 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 428.8359375) proc.memory.percent‚17.18275547027588* disk./.usagePercent‚36.828889817388394 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.0065229e+07 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚144824.6953125$ memory_percent‚97.30696678161621 disk.drbd2165.in‚0 disk.dm-1.out‚ 428.8359375" disk.dm-0.out‚52757.1982421875 cpu‚26.03373744624627 proc.cpu.threads‚17' disk.nvme0c0n1.out‚52757.1982421875Q†A“–(‚(:ˆ ׃èÒ•¥«¢ gpu.0.gpu‚95.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}g—(‚(:ú ׃èÒ½ÍΤ disk.rbd0.out‚0$ disk./.usageGB‚548.7770195007324! disk.dm-0.out‚52763.380859375$ disk.nvme0n1.out‚52763.380859375' proc.memory.availableMB‚ 441.2890625 disk.dm-0.in‚144825.41015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 429.3203125 proc.cpu.threads‚17* disk./.usagePercent‚36.828886233282915 disk.rbd3.in‚0 disk.rbd0.in‚0! disk.rbd1.out‚111.16845703125# proc.memory.rssMB‚ 2815.22265625 cpu‚26.034106872562358 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda.out‚ 429.3203125 network.sent‚ 2.0128549e+07 network.recv‚ 2.889855e+06 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚52763.380859375 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚97.30658531188965# disk.nvme0n1.in‚144825.41015625 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144825.41015625 disk.sda.in‚0 disk.sda3.out‚ 429.3203125) proc.memory.percent‚17.18275547027588 disk.sda3.in‚0 disk.rbd1.in‚0.0625JTñp4˜(/ _wandbruntime‚10812  _runtime‚10812í ¦Ì™(Æ ' lr-AdamW/pg1‚0.00014418089508967032 ' lr-AdamW/pg2‚0.00014418089508967032 trainer/global_step‚12699 % _timestamp‚1.7842836203667498e+09 _runtime‚10812.492385613 _step‚1043n=¬àÑš(Ë _runtime‚10812.492385613 _step‚1043 ' lr-AdamW/pg1‚0.00014418089508967032 ' lr-AdamW/pg2‚0.00014418089508967032 trainer/global_step‚12699 % _timestamp‚1.7842836203667498e+09“¾FÃõ4›(/ _wandbruntime‚10812  _runtime‚10812Ƹëþœ(P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100670908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13406489797398 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1929813483470848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1919889376227328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1943240444280832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1929813483470848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13576693844480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1943240444280832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13427319188480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117607650 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27324593 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100670906 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842836203678055e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1943240802659328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1929813466431488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218288535 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚12699 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117617627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22245501 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13427319188480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1933296206574616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1929813466431488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1919889359187968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22245498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚10812.493588174 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27324858 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13426977849344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚438739073932800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117607650 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269ÝÉÅÞ ((†P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22245498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚10812.493588174 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27324858 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13426977849344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚438739073932800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117607650 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100670908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13406489797398 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1929813483470848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1919889376227328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1943240444280832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1929813483470848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13576693844480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1943240444280832 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13427319188480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117607650 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27324593 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100670906 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842836203678055e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1943240802659328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1929813466431488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218288535 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚12699 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117617627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22245501 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13427319188480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1933296206574616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1929813466431488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1919889359187968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169”ëÄ9I4ž(/ _wandbruntime‚10813  _runtime‚10813ã®Î'Ÿ(ŠN TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚438773628422144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117616702 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49574160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1943388697937408 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218305233 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100678538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100678540 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1943389056323072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218295240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117626693 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13577712386048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1943389056323072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1929960724500480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218305233 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49573879 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1943389056323072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218295240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117626693 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13577712386048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1943389056323072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1929960724500480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218305233 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49573879 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚438773628422144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117616702 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49574160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1943388697937408 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218305233 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100678538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100678540 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1950653549642752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118072079 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1950653191264256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27428764 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1937175534952448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1937175517913088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101052508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101052506 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13627836521472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219114608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22332751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426833925203968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1937175534952448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118062102 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚426833906591744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1950653191264256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219124587 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚12749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1927214892781568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1927214909820928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1950653549642752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118072079 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1950653191264256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27428764 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1937175534952448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1937175517913088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101052508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101052506 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13627836521472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219114608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22332751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426833925203968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1937175534952448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118062102 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚426833906591744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1950653191264256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219124587 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚12749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1927214892781568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1927214909820928 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219141285 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101060138 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1950801444920832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22334496 _step‚1049 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219131292 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101060140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118071154 PBDeviceStatsMonitor.on_train_batcÓ)7Ý!h_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13479027324416 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49765001 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1950801803306496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118071154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22334493 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101060140 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13458117338028 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13458457888030 % _timestamp‚1.7842836614187555e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440496094462464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚426867353962496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1940819520791468 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1927361403453440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13478685978112 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13478685978112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440496205838848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1927361420492800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13628851876352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101060138 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1937322758942720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27430775 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27430508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚426867335350272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219131292 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1937322775982080 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1937322775982080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13479027324416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12749 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1940819878380830 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118081145 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1950801803306496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118081145 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219141285 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13628759112192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Vúz'²(N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1940819878380830 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118081145 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1950801803306496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118081145 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219141285 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13628759112192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219141285 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101060138 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1950801444920832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22334496 _step‚1049 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219131292 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101060140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118071154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13479027324416 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49765001 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1950801803306496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118071154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22334493 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101060140 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13458117338028 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13458457888030 % _timestamp‚1.7842836614187555e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440496094462464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚426867353962496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1940819520791468 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1927361403453440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13478685978112 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13478685978112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440496205838848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1927361420492800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13628851876352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101060138 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1937322758942720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27430775 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27430508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚426867335350272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219131292 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1937322775982080 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1937322775982080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13479027324416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12749 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0™™#yé4³(/ _wandbruntime‚10853  _runtime‚10853j›Œó´(Š 2loss/SRqqt1/assignment_loss‚0.3820628896625727 3loss/SRqqt2/assignment_loss‚0.47941859009169346 /loss/FBt2/detection_loss‚0.8474151814589277 % _timestamp‚1.7842836614219067e+09 _runtime‚10853.547259941 0loss/FRt2/assignment_loss‚0.8634713529324045 1loss/SRqqt2/detection_loss‚0.6105878703870259 /loss/FBt1/detection_loss‚0.4535890880474863 epoch‚31 2loss/FBt1/assignment_loss‚0.015493790953567118 /loss/FRt1/detection_loss‚0.5104683785754656 /loss/FRt2/detection_loss‚1.2350165992153597 %loss/total_loss‚7.914390240108254 trainer/global_step‚12749 _step‚1050 4loss/FBt2/assignment_loss‚0.00014379608084357689 1loss/SRqqt1/detection_loss‚0.8787723064618019 0loss/FRt1/assignment_loss‚1.6379503962411062î%µ( 0loss/FRt1/assignment_loss‚1.6379503962411062 2loss/SRqqt1/assignment_loss‚0.3820628896625727 3loss/SRqqt2/assignment_loss‚0.47941859009169346 /loss/FBt2/detection_loss‚0.8474151814589277 % _timestamp‚1.7842836614219067e+09 _runtime‚10853.547259941 0loss/FRt2/assignment_loss‚0.8634713529324045 1loss/SRqqt2/detection_loss‚0.6105878703870259 /loss/FBt1/detection_loss‚0.4535890880474863 epoch‚31 2loss/FBt1/assignment_loss‚0.015493790953567118 /loss/FRt1/detection_loss‚0.5104683785754656 /loss/FRt2/detection_loss‚1.2350165992153597 %loss/total_loss‚7.914390240108254 trainer/global_step‚12749 _step‚1050 4loss/FBt2/assignment_loss‚0.00014379608084357689 1loss/SRqqt1/detection_loss‚0.8787723064618019šA/S%“¶(‚(:ˆ “„èÒ²ŽÆ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿz×8ý·(‚(:ò “„èÒéó­¥ network.recv‚ 2.904078e+06 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚26.0357953026303% disk.nvme0n1.out‚52794.5634765625 disk.sda1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚52794.5634765625 disk.dm-1.out‚ 430.2109375 disk.rbd3.out‚0 disk.nvme0n1.in‚ 144827.78125 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.rbd1.out‚111.38720703125 disk.rbd3.in‚0 disk.dm-0.in‚ 144827.78125" disk.dm-0.out‚52794.5634765625 network.sent‚ 2.0206899e+07 disk.sda.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.30839729309082# proc.memory.rssMB‚ 2815.22265625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 144827.78125 disk.sda.out‚ 430.2109375 disk.dm-1.in‚0* disk./.usagePercent‚36.828914650119216 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 430.2109375' proc.memory.availableMB‚ 440.9921875) proc.memory.percent‚17.18275547027588$ disk./.usageGB‚548.7774429321289-æ “¸(‚(:ˆ ¢„èÒíÉ¿¢ gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚260.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D¥`¹(‚(:ƒ ¢„èÒÙǸ¥ disk.sda3.in‚0 disk.dm-1.out‚ 430.34765625! disk.rbd1.out‚111.38720703125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚144828.49609375# disk.dm-0.out‚52803.90576171875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚52803.90576171875 disk.sda.out‚ 430.34765625 disk.rbd0.in‚0( proc.memory.availableMB‚ 440.78515625) proc.memory.percent‚17.18275547027588 cpu‚26.036249365067544 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0# proc.memory.rssMB‚ 2815.22265625* disk./.usagePercent‚36.828912858066474# disk./.usageGB‚548.777416229248 disk.sda.in‚0 disk.sda2.in‚0# disk.nvme0n1.in‚144828.49609375& disk.nvme0n1.out‚52803.90576171875 disk.sda1.out‚0 disk.drbd2165.out‚0 network.sent‚ 2.0269566e+07% disk.nvme0c0n1.in‚144828.49609375 disk.dm-1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 430.34765625$ memory_percent‚97.30966091156006 network.recv‚ 2.908274e+06i‹‡bžº(‚(:“ ±„èÒ—ãǤ gpu.0.gpu‚45.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µêa»(‚(:÷ ±„èÒ‡˜æ¥ disk.drbd2165.in‚0 disk.sda3.out‚ 430.78125# proc.memory.rssMB‚ 2815.22265625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 430.78125! disk.rbd1.out‚111.44970703125# disk.nvme0n1.in‚144828.49609375% disk.nvme0n1.out‚52806.7646484375 disk.sda1.out‚0 disk.sda.out‚ 430.78125 disk.rbd1.in‚0.0625 proc.cpu.threads‚17 network.sent‚ 2.0278219e+07 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚52806.7646484375$ disk./.usageGB‚548.7774620056152' disk.nvme0c0n1.out‚52806.7646484375 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-0.in‚144828.49609375$ memory_percent‚97.30675220489502( proc.memory.availableMB‚ 441.26171875 network.recv‚ 2.918681e+06% disk.nvme0c0n1.in‚144828.49609375) proc.memory.percent‚17.18275547027588) disk./.usagePercent‚36.82891593015689 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚26.036677939068397rŠG\4¼(/ _wandbruntime‚10893  _runtime‚10893Xóflʽ(Ä & lr-AdamW/pg2‚0.0001407046370479589 trainer/global_step‚12799 % _timestamp‚1.7842837009857397e+09 _runtime‚10893.112335414 _step‚1051 & lr-AdamW/pg1‚0.0001407046370479589ÅÙyϾ(É trainer/global_step‚12799 % _timestamp‚1.7842837009857397e+09 _runtime‚10893.112335414 _step‚1051 & lr-AdamW/pg1‚0.0001407046370479589 & lr-AdamW/pg2‚0.0001407046370479589›†˜uÙ4¿(/ _wandbruntime‚10893  _runtime‚108933é 4À(P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13528710192128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27532088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428505344518144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1052 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1958066296626176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219960639 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚12799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118526531 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13528368852992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1934540443414528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13678815942144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13528368852992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚442184273940992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13528710192128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101434108 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101434106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1948048149146382 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13507722771214 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13508063321104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22419998 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13678910810624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1944537569394688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1944537569394688 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118516554 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1958065938247680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1944537586434048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49952089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚442184160460288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1958065938247680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101434106 IÆÆ›Ó UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚10893.112440673 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219950660 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101434108 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1958066296626176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27531823 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118526531 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1934540426375168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22420001Uø%J (Á(†P GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1948048149146382 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13507722771214 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13508063321104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22419998 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13678910810624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1944537569394688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1944537569394688 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118516554 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1958065938247680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1944537586434048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49952089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚442184160460288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1958065938247680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101434106 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚10893.112440673 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219950660 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101434108 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1958066296626176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27531823 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118526531 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1934540426375168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22420001 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13528710192128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27532088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428505344518144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1052 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1958066296626176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219960639 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚12799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118526531 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13528368852992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1934540443414528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13678815942144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13528368852992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚442184273940992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13528710192128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101434108 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101434106œç/Ü4Â(/ _wandbruntime‚10893  _runtime‚10893ƒ„÷˜'Ã(ŠN C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚10893.723395397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118535597 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13529381479936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219977337 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118535597 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1944684827463680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13529722826240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13509074374938 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13529381479936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219977337 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27533860 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118525606 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428538773276672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784283701597495e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118525606 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13679925682176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1934686954086400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219967344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49955870 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1944684827463680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1948195670871976 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13508733824936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚442218606194688 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49955603 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚442218717571072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428538791888896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101441740 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22421746 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13529722826240 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1948196028461338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1958214191904256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22421743 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101441740 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0( ßÚº&Ä(N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219967344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49955603 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚442218717571072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428538791888896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101441740 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22421746 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13529722826240 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1948196028461338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1958214191904256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22421743 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101441740 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚10893.723395397 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118535597 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13529381479936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219977337 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118535597 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1944684827463680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13529722826240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13509074374938 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13529381479936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219977337 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27533860 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118525606 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428538773276672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 $ _timestamp‚1.784283701597495e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118525606 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13679925682176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1934686954086400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219967344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49955870 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1944684827463680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1948195670871976 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13508733824936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚442218606194688 F&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6612433195114136 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7517762184143066 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18090488016605377 particle/accuracy_1_of_3‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26592668890953064 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_4_of_5‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.730461061000824 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6293905973434448 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7825655341148376 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_3‚0.9331514239311218 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6648001670837402 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6096354126930237 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7875882387161255 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_3_of_5‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148655891418 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6515906453132629 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049836255609989166 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5258333683013916 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5046517848968506 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7931913733482361 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8989146947860718 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6300205588340759 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +particle/specificity‚0.9053905010223389 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2701131999492645 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2075374722480774 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7690197229385376 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875767827033997 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2355749011039734 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22748106718063354 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2520880401134491 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16083332896232605 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27379927039146423 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7407407164573669 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5640021562576294 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6192132830619812 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988857090473175 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021230366080999374 (particle/accuracy‚0.8439554572105408 _runtime‚10910.81365559 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04962713271379471 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6238183379173279 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6375593543052673 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26226064562797546 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3533988296985626 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6835694313049316 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6349247694015503 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016975030303001404 7validation_average_jet_accuracy‚0.39254528284072876 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995618462562561 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7361280918121338 *jet/accuracy_2_of_2‚0.1638784110546112 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6727323532104492 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.780532956123352 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35413432121276855 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19599321484565735 +particle/sensitivity‚0.6192887425422668 'particle/f_score‚0.6302192211151123 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 $ _timestamp‚1.784283718687242e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6260278224945068 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30180448293685913 *jet/accuracy_2_of_3‚0.5820589661598206 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.665045976638794 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7013583779335022 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8652123808860779 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980031847953796 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6400237083435059 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5144185423851013 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6999221444129944 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.023020043969154358 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8652123808860779 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3109247088432312 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.775360643863678 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32538995146751404 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7365834712982178 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30180448293685913 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9907498359680176 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24529221653938293 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5907590985298157 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7203397154808044 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0824424996972084 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2932587265968323 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3789747357368469 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35790371894836426 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8496440052986145 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3780176639556885 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02107234299182892 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03983457013964653 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.803691029548645 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.192713662981987 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.25001755356788635 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8018316626548767 +jet/accuracy_3_of_4‚0.07939416915178299 particle/accuracy_1_of_1‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2397385537624359 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26164481043815613 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚1055 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970593452453613 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5433777570724487 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2723223567008972 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ,jet/accuracy_4_of_4‚0.011098379269242287 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6897101402282715 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_3‚0.0631408840417862 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7392479181289673 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5594686269760132 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4115477204322815 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5752155184745789 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 trainer/global_step‚12799 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09463130682706833 *jet/accuracy_1_of_4‚0.9765217304229736 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994560956954956 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2598246932029724 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_4_of_4‚0.9535390138626099 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9104152917861938 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.854577362537384 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.705161452293396 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7052086591720581 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_5‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7237717509269714 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022812839597463608 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30136528611183167 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7231873869895935 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.632355809211731 particle/accuracy_2_of_2‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7278990745544434 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4726274907588959 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5241036415100098 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29245680570602417 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 -"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3065266013145447 )loss/val_total_loss‚7.994249825653797 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6629182696342468 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9887582063674927 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 epoch‚31 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6860048770904541 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6375593543052673 particle/accuracy_5_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6556522250175476 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04347013682126999 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7407407164573669 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.36704799532890323@¢K±5Ì(ˆ{ /particle/accuracy_4_of_4‚0.9535390138626099 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9104152917861938 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.854577362537384 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.705161452293396 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7052086591720581 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_5‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7237717509269714 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022812839597463608 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30136528611183167 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7231873869895935 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.632355809211731 particle/accuracy_2_of_2‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7278990745544434 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4726274907588959 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5241036415100098 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29245680570602417 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 -"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3065266013145447 )loss/val_total_loss‚7.994249825653797 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6629182696342468 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9887582063674927 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 epoch‚31 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6860048770904541 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6375593543052673 particle/accuracy_5_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6556522250175476 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04347013682126999 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7407407164573669 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016975030303001404 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7345770001411438 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_1_of_4‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31071650981903076 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31102705001831055 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6612433195114136 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7517762184143066 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18090488016605377 particle/accuracy_1_of_3‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26592668890953064 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_4_of_5‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.730461061000824 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6293905973434448 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7825655341148376 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_3‚0.9331514239311218 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6648001670837402 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6096354126930237 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7875882387161255 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_3_of_5‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148655891418 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6515906453132629 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.049836255609989166 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5258333683013916 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5046517848968506 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7931913733482361 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8989146947860718 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6300205588340759 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +particle/specificity‚0.9053905010223389 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2701131999492645 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 jet/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2075374722480774 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7690197229385376 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875767827033997 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2355749011039734 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22748106718063354 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2520880401134491 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16083332896232605 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27379927039146423 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7407407164573669 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5640021562576294 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6192132830619812 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988857090473175 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021230366080999374 (particle/accuracy‚0.8439554572105408 _runtime‚10910.81365559 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04962713271379471 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6238183379173279 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6375593543052673 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26226064562797546 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3533988296985626 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6835694313049316 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6349247694015503 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016975030303001404 7validation_average_jet_accuracy‚0.39254528284072876 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995618462562561 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7361280918121338 *jet/accuracy_2_of_2‚0.1638784110546112 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6727323532104492 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.780532956123352 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35413432121276855 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19599321484565735 +particle/sensitivity‚0.6192887425422668 'particle/f_score‚0.6302192211151123 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 $ _timestamp‚1.784283718687242e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6260278224945068 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30180448293685913 *jet/accuracy_2_of_3‚0.5820589661598206 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.665045976638794 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7013583779335022 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8652123808860779 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980031847953796 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6400237083435059 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5144185423851013 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6999221444129944 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.023020043969154358 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8652123808860779 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3109247088432312 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.775360643863678 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32538995146751404 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7365834712982178 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30180448293685913 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9907498359680176 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24529221653938293 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5907590985298157 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7203397154808044 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0824424996972084 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2932587265968323 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3789747357368469 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35790371894836426 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8496440052986145 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3780176639556885 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02107234299182892 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03983457013964653 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.803691029548645 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.192713662981987 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.25001755356788635 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8018316626548767 +jet/accuracy_3_of_4‚0.07939416915178299 particle/accuracy_1_of_1‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt§±(Ý*SRqqt0FBt/FRt_purity‚0.2397385537624359 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26164481043815613 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚1055 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970593452453613 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5433777570724487 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2723223567008972 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ,jet/accuracy_4_of_4‚0.011098379269242287 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6897101402282715 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_3‚0.0631408840417862 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7392479181289673 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5594686269760132 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4115477204322815 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5752155184745789 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 trainer/global_step‚12799 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09463130682706833 *jet/accuracy_1_of_4‚0.9765217304229736 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994560956954956 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2598246932029724 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837Ÿ|° ’Í(‚(:‡ Ï„èÒæ½£ gpu.0.gpu‚23.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¥øúÎ(‚(:ï Ï„èÒÚË⤠disk.dm-1.in‚0 network.recv‚ 2.926667e+06 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚144830.0390625( proc.memory.availableMB‚ 436.80078125" disk.nvme0n1.in‚144830.0390625 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 network.sent‚ 2.034648e+07$ disk.nvme0c0n1.in‚144830.0390625 disk.sda.in‚0! disk.rbd1.out‚111.51220703125$ disk.nvme0n1.out‚52822.806640625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚52822.806640625 disk.sda.out‚ 431.34375 disk.sda3.in‚0 disk.sda3.out‚ 431.34375! disk.dm-0.out‚52822.806640625) disk./.usagePercent‚36.82880584691716# disk./.usageGB‚548.775821685791$ memory_percent‚97.33397960662842 disk.drbd2165.in‚0* proc.memory.percent‚17.206907272338867 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 431.34375" proc.memory.rssMB‚ 2819.1796875 cpu‚26.042145550139644 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0&ÖÎ ‘Ï(‚(:† Þ„èÒƒ£¶¢ gpu.0.gpu‚28.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ³MCýÐ(‚(:ò Þ„è҉ܤ$ disk./.usageGB‚548.7758636474609 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 431.53125 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 431.53125& proc.memory.availableMB‚ 436.796875 network.sent‚ 2.0403701e+07) disk./.usagePercent‚36.82880866300004 disk.sda.out‚ 431.53125" disk.dm-0.out‚52831.7431640625* proc.memory.percent‚17.206907272338867 cpu‚26.041701183101118 disk.dm-1.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚111.51220703125 network.recv‚ 2.931014e+06" disk.nvme0n1.in‚144830.8671875$ disk.nvme0c0n1.in‚144830.8671875' disk.nvme0c0n1.out‚52831.7431640625 disk.sda2.out‚0" proc.memory.rssMB‚ 2819.1796875% disk.nvme0n1.out‚52831.7431640625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚144830.8671875$ memory_percent‚97.33400344848633 proc.cpu.threads‚17£5C;Ñ(‚(:’ í„èÒᛓ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚283.318' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.66359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?8NXÒ(‚(:ý í„èÒ¶ºÌ¥" disk.nvme0n1.in‚144830.8671875% disk.nvme0n1.out‚52835.4638671875 disk.drbd2165.out‚0 disk.sda.in‚0* proc.memory.percent‚17.206907272338867 network.recv‚ 2.934618e+06" disk.dm-0.out‚52835.4638671875$ disk.nvme0c0n1.in‚144830.8671875( proc.memory.availableMB‚ 436.79296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚52835.4638671875 disk.sda2.in‚0 disk.sda3.out‚ 431.81640625 disk.rbd3.in‚0$ disk./.usageGB‚548.7760200500488 disk.rbd1.in‚0.0625 proc.cpu.threads‚17) disk./.usagePercent‚36.82881915930894 disk.drbd2165.in‚0 disk.sda.out‚ 431.81640625" proc.memory.rssMB‚ 2819.1796875 network.sent‚ 2.0408745e+07 disk.rbd0.out‚0 disk.dm-1.in‚0! disk.rbd1.out‚111.57470703125$ memory_percent‚97.33402729034424 disk.sda2.out‚0 disk.dm-1.out‚ 431.81640625 disk.rbd3.out‚0 cpu‚26.041299010370928 disk.dm-0.in‚144830.8671875 disk.sda3.in‚09tçí4Ó(/ _wandbruntime‚10950  _runtime‚10950´÷ùuÊÔ(Ä & lr-AdamW/pg2‚0.0001389756654587799 trainer/global_step‚12849 % _timestamp‚1.7842837580288177e+09 _runtime‚10950.154390968 _step‚1056 & lr-AdamW/pg1‚0.0001389756654587799ØEÏÕ(É & lr-AdamW/pg2‚0.0001389756654587799 trainer/global_step‚12849 % _timestamp‚1.7842837580288177e+09 _runtime‚10950.154390968 _step‚1056 & lr-AdamW/pg1‚0.0001389756654587799 dEÀ¹4Ö(/ _wandbruntime‚10950  _runtime‚10950Ì•ÎZ(×(€P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101936372 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101936372 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22509771 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119041328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13597138899456 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50176855 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27667084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101936374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1953829204566016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13748307942912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1967425985086976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22509768 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842837580298202e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1967426343465472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50176604 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13748240337408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13576058983351 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13596797560320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119041328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220987679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67605504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1943774623182848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220977700 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1953829187526656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101936374»¾KÇ (Ø(…P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50176604 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13748240337408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13576058983351 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13596797560320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119041328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220987679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67605504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1943774623182848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220977700 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1953829187526656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101936374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101936372 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101936372 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22509771 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119041328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13597138899456 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50176855 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27667084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101936374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1953829204566016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13748307942912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1967425985086976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22509768 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842837580298202e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1967426343465472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22511516 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13577410587075 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1957498203891793 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13597810187264 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1943921133854720 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101944004 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119050380 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1967574597129216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚10950.76729381 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430226123302912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1953976428556288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13577070037073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMon yêZitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50180437 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101944006 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50180689 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220994384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221004377 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22511513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443975392867328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1953976445595648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220994384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12849 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1967574238743552 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1967574597129216 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13597810187264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13749269564416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1953976428556288 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13598151533568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27669173 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 % _timestamp‚1.7842837586414664e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27668924 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1957498561481155θ’|'Û(ŽN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1953976445595648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220994384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12849 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1967574238743552 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1967574597129216 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13597810187264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13749269564416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1953976428556288 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13598151533568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27669173 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 % _timestamp‚1.7842837586414664e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27668924 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1957498561481155 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚443975491660800 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22511516 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13577410587075 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1957498203891793 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13597810187264 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1943921133854720 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101944004 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119050380 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1967574597129216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚10950.76729381 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430226123302912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1953976428556288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13577070037073 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50180437 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101944006 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50180689 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220994384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221004377 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22511513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443975392867328 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438¢Þ'Ž4Ü(/ _wandbruntime‚10950  _runtime‚10950Ïâe Ý(„ 2loss/SRqqt2/assignment_loss‚0.4875882433566484 1loss/SRqqt1/detection_loss‚0.8942885928929014 epoch‚32 _runtime‚10950.77025484 0loss/FRt1/assignment_loss‚1.4809726327488175 .loss/FRt2/detection_loss‚1.162637720541194 /loss/FBt1/detection_loss‚0.4109268666908523 trainer/global_step‚12849 $ _timestamp‚1.784283758644524e+09 _step‚1059 0loss/FRt2/assignment_loss‚0.7815713667238593 2loss/SRqqt1/assignment_loss‚0.3452269532108324 3loss/FBt1/assignment_loss‚0.0030323654403671754 3loss/FBt2/assignment_loss‚0.0035375677850567376 .loss/FRt1/detection_loss‚0.513519370693308 1loss/SRqqt2/detection_loss‚0.5384177341243026 .loss/FBt2/detection_loss‚0.763461007567292 %loss/total_loss‚7.385180421775431âMi<Þ(‰ 0loss/FRt1/assignment_loss‚1.4809726327488175 .loss/FRt2/detection_loss‚1.162637720541194 /loss/FBt1/detection_loss‚0.4109268666908523 trainer/global_step‚12849 $ _timestamp‚1.784283758644524e+09 _step‚1059 0loss/FRt2/assignment_loss‚0.7815713667238593 2loss/SRqqt1/assignment_loss‚0.3452269532108324 3loss/FBt1/assignment_loss‚0.0030323654403671754 3loss/FBt2/assignment_loss‚0.0035375677850567376 .loss/FRt1/detection_loss‚0.513519370693308 1loss/SRqqt2/detection_loss‚0.5384177341243026 .loss/FBt2/detection_loss‚0.763461007567292 %loss/total_loss‚7.385180421775431 2loss/SRqqt2/assignment_loss‚0.4875882433566484 1loss/SRqqt1/detection_loss‚0.8942885928929014 epoch‚32 _runtime‚10950.77025484£Å"Û¾“ß(‚(:ˆ ü„èÒ®¯é¢ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚237.593' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5186 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªNýŽ à(‚(:‚ ü„èÒÙ—ê¦ disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 432.17578125 network.sent‚ 2.0426833e+07) disk./.usagePercent‚36.82893564273702 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚144831.58203125 disk.drbd2165.out‚0 disk.sda.out‚ 432.17578125 cpu‚26.040850632152218 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.206907272338867 network.recv‚ 2.938238e+06 disk.rbd0.in‚0 disk.sda3.out‚ 432.17578125$ memory_percent‚97.33548164367676$ disk./.usageGB‚548.7777557373047% disk.nvme0c0n1.in‚144831.58203125( disk.nvme0c0n1.out‚52847.21533203125 disk.rbd1.in‚0.0625! disk.rbd1.out‚111.57470703125 disk.rbd3.out‚0# disk.dm-0.out‚52847.21533203125' proc.memory.availableMB‚ 436.5546875" proc.memory.rssMB‚ 2819.1796875# disk.nvme0n1.in‚144831.58203125& disk.nvme0n1.out‚52847.21533203125œœDà“á(‚(:ˆ ‹…èÒÑš³¢ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚257.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ÜéK â(‚(:ÿ ‹…èÒøü˜¤$ disk./.usageGB‚548.7775421142578 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2819.1796875( disk.nvme0c0n1.out‚52853.76708984375 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚144832.41015625# disk.nvme0n1.in‚144832.41015625 disk.dm-1.out‚ 432.4296875 proc.cpu.threads‚17 network.sent‚ 2.0476827e+07 disk.sda.out‚ 432.4296875 cpu‚26.040422519365293 disk.sda3.in‚0 disk.sda3.out‚ 432.4296875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0% disk.nvme0c0n1.in‚144832.41015625 disk.drbd2165.in‚0$ memory_percent‚97.33359813690186( proc.memory.availableMB‚ 436.86328125# disk.dm-0.out‚52853.76708984375 disk.rbd0.in‚0! disk.rbd1.out‚111.69970703125* proc.memory.percent‚17.206907272338867( disk./.usagePercent‚36.8289213063151& disk.nvme0n1.out‚52853.76708984375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 network.recv‚ 2.942849e+06õëkW4ã(/ _wandbruntime‚10989  _runtime‚10989žºLdÌä(Æ trainer/global_step‚12899 % _timestamp‚1.7842837977233136e+09 _runtime‚10989.848908353 _step‚1060 ' lr-AdamW/pg1‚0.00013725294702036963 ' lr-AdamW/pg2‚0.00013725294702036963˜ · Ñå(Ë ' lr-AdamW/pg1‚0.00013725294702036963 ' lr-AdamW/pg2‚0.00013725294702036963 trainer/global_step‚12899 % _timestamp‚1.7842837977233136e+09 _runtime‚10989.848908353 _step‚1060¤¨ÚL4æ(/ _wandbruntime‚10989  _runtime‚10989º*|I(ç(P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102317974 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27772468 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1961191239008256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102317972 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431864132470784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842837977244256e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119505757 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22597021 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1951100173815808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 221823731 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445663820773888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13799587405312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1974838732070400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13627016020149 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102317972 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445663719876096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚10989.850198619 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1961191256047616 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1961191239008256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50369739 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1951100156776448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13647834401280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27772718 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1974839090448896 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1961191256047616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13647493062144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13626675470259 _step‚1061 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1964726832246707 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13799669690880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1974839090448896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1974838732070400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119495780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22597018 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚12899 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13647834401280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 221813752 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119495780 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1964727189835957 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50369486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0·/“ è(†P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13627016020149 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102317972 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445663719876096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚10989.850198619 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1961191256047616 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1961191239008256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50369739 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1951100156776448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13647834401280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27772718 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1974839090448896 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1961191256047616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13647493062144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13626675470259 _step‚1061 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1964726832246707 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13799669690880 E?DeviceStatsMonitor.on_train_batch_Š‚xystart/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1974839090448896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1974838732070400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119495780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22597018 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚12899 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13647834401280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 221813752 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119495780 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1964727189835957 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50369486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102317974 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27772468 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1961191239008256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102317972 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431864132470784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842837977244256e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119505757 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22597021 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1951100173815808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 221823731 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445663820773888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13799587405312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1974838732070400¥‚îâ24é(/ _wandbruntime‚10990  _runtime‚10990IŸ õ'ê(ŠN SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13648847035392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1951246684487680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221830436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22598766 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27774805 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13627686523981 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13648847035392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445698183409152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50373571 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221840429 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1961338497077248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102325604 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13800622179840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50373303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22598763 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12899 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119504832 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1974987344112640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27774540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1961338480037888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119504832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221840429 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445698282202624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431897561229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119514823 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1964874711561663 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1961338497077248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13628027073983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102325606 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1974987344112640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13648505689088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12899 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119504832 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1974987344112640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27774540 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1961338480037888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119504832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221840429 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445698282202624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431897561229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119514823 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1964874711561663 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1961338497077248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13628027073983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102325606 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1974987344112640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13648505689088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27774805 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13627686523981 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13648847035392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445698183409152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50373571 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221840429 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1961338497077248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102325604 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13800622179840 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50373303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22598763 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842838376386862e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27877696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1958425690370048 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚11029.764465735 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102699574 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13850844609536 _step‚1065 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13698529903104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50562245 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22684271 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27877974 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1972102982327215 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13677632507057 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222659783 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119960209 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1968553290489856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1982251479053824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50561964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1968553290489856 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119960209 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚447386415006720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚433535570397184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1972103339916465 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119950232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119950232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚447386520098816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1982251479053824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13850931089408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13698188563968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚433535589009408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13698529903104 trainer/global_step‚12949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222649804 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1958425707409408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269«eÖ¹ (ú(†P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1982251479053824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50561964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1968553290489856 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119960209 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚447386415006720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚433535570397184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1972103339916465 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119950232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119950232 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚447386520098816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1982251479053824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13850931089408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13698188563968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚433535589009408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13698529903104 trainer/global_step‚12949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222649804 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1958425707409408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1968553307529216 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1982251837432320 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102699572 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842838376386862e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27877696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1958425690370048 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚11029.764465735 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102699574 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13850844609536 _step‚1065 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13698529903104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50562245 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22684271 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27877974 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1972102982327215 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13677632507057 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222659783 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119960209 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1968553290489856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0©+™d¨4û(/ _wandbruntime‚11030  _runtime‚11030Í;&î'ü(‹N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222666488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1982399732710400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1968700548558848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 trainer/global_step‚12949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102707206 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50565783 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13699542537216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222676481 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102707204 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842838382497723e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50566044 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102707204 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447420967644672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222676481 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102707206 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119969275 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119959284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22686013 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1972250504052809 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13699542537216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222676481 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102707204 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842838382497723e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50566044 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102707204 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447420967644672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222676481 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102707206 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119969275 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119959284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22686013 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1972250504052809 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50565783 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1989664226037248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27983044 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1989664584415744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50754816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842838776098201e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120414661 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435207008323584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1975915341971456 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1979479132407723 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120404684 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13748884065792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22771518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚435207026935808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13727908444075 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103081172 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223495835 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1965751241003008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50754562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1975915359010816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13728248993965 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976îˆV4 Š)†P JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223495835 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1965751241003008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50754562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1975915359010816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13728248993965 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1989664584415744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13749225404928 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1989664226037248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27983044 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1989664584415744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50754816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842838776098201e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120414661V2Æö aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚435207008323584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1975915341971456 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1979479132407723 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120404684 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13748884065792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22771518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚435207026935808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13727908444075 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103081172 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1976062600040448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103088804 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13728919497797 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13903065605120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1989812479693824 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1979627011722679 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449143502687232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435240437082112 _step‚1070 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1976062583001088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1976062583001088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223512533 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223502540 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13749896692736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12999 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1965897751674880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223502540 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120423727 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13729260047799 % _timestamp‚1.7842838782198334e+09 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50758297 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435240455694336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1979626654133317 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223512533 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103088804 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1989812838079488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13750238039040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13749896692736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22773266 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13903065605120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1989812479693824 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1979627011722679 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449143502687232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435240437082112 _step‚1070 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1976062583001088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1976062583001088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223512533 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223502540 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13749896692736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12999 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1965897751674880 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223502540 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120423727 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13729260047799 % _timestamp‚1.7842838782198334e+09 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50758297 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435240455694336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1979626654133317 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223512533 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103088804 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1989812838079488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13750238039040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13749896692736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22773266 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1976062600040448 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103088804®íCÐ 4)/ _wandbruntime‚11070  _runtime‚11070âž‘)‹ &loss/total_loss‚7.3695639891155125 0loss/FRt1/assignment_loss‚1.5555266380442219 3loss/SRqqt1/assignment_loss‚0.37263816531978555 3loss/FBt1/assignment_loss‚0.0009760027789231055 3loss/FBt2/assignment_loss‚9.065520453303706e-05 .loss/FRt2/detection_loss‚1.245022406971388 3loss/SRqqt2/assignment_loss‚0.40552472662424655 /loss/FBt1/detection_loss‚0.3364518791508999 epoch‚32 trainer/global_step‚12999 1loss/SRqqt2/detection_loss‚0.5823646954353898 /loss/FBt2/detection_loss‚0.5913465242219073 0loss/FRt2/assignment_loss‚0.7981164376710502 /loss/FRt1/detection_loss‚0.5462316138411568 1loss/SRqqt1/detection_loss‚0.9352742438520104 _step‚1071 % _timestamp‚1.7842838782234704e+09 _runtime‚11070.348803429…,Ó’) /loss/FRt1/detection_loss‚0.5462316138411568 1loss/SRqqt1/detection_loss‚0.9352742438520104 _step‚1071 % _timestamp‚1.7842838782234704e+09 _runtime‚11070.348803429 &loss/total_loss‚7.3695639891155125 0loss/FRt1/assignment_loss‚1.5555266380442219 3loss/SRqqt1/assignment_loss‚0.37263816531978555 3loss/FBt1/assignment_loss‚0.0009760027789231055 3loss/FBt2/assignment_loss‚9.065520453303706e-05 .loss/FRt2/detection_loss‚1.245022406971388 3loss/SRqqt2/assignment_loss‚0.40552472662424655 /loss/FBt1/detection_loss‚0.3364518791508999 epoch‚32 trainer/global_step‚12999 1loss/SRqqt2/detection_loss‚0.5823646954353898 /loss/FBt2/detection_loss‚0.5913465242219073 0loss/FRt2/assignment_loss‚0.7981164376710502¯Ðfý“)‚(:’ ô…èÒúù°£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚292.548' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.50959999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å½æy ”)‚(:þ ô…èÒ´”©¥$ memory_percent‚97.33612537384033" proc.memory.rssMB‚ 2819.1796875* proc.memory.percent‚17.206907272338867 disk.rbd1.in‚0.0625 disk.rbd3.out‚0# disk.dm-0.out‚52904.03173828125 proc.cpu.threads‚17 network.sent‚ 2.0674904e+07 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚144836.2109375( disk.nvme0c0n1.out‚52904.03173828125 disk.sda3.out‚ 434.1015625 disk.rbd3.in‚0& disk.nvme0n1.out‚52904.03173828125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 436.44921875 disk.sda2.out‚0 disk.dm-0.in‚144836.2109375 network.recv‚ 2.969438e+06 disk.drbd2165.in‚0 disk.dm-1.out‚ 434.1015625$ disk./.usageGB‚548.7774124145508 cpu‚26.037430239626588" disk.nvme0n1.in‚144836.2109375 disk.sda1.out‚0 disk.sda.out‚ 434.1015625 disk.dm-1.in‚0* disk./.usagePercent‚36.828912602058935 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0! disk.rbd1.out‚111.88720703125FË ‡“•)‚(:ˆ ƒ†èÒœœž¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0܉‹%–)‚(:ù ƒ†èÒô¾¤( disk.nvme0c0n1.out‚52910.12158203125$ disk.nvme0c0n1.in‚144837.0390625 disk.sda2.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2819.1796875 network.recv‚ 2.973692e+06 disk.rbd0.in‚0( proc.memory.availableMB‚ 436.72265625 disk.rbd3.out‚0) disk./.usagePercent‚36.82893001057127 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚548.7776718139648 disk.rbd0.out‚0! disk.rbd1.out‚111.94970703125 disk.rbd1.in‚0.0625 disk.dm-0.in‚144837.0390625# disk.dm-0.out‚52910.12158203125 disk.sda3.out‚ 434.453125 disk.rbd3.in‚0 disk.sda.out‚ 434.453125 disk.dm-1.out‚ 434.453125" disk.nvme0n1.in‚144837.0390625& disk.nvme0n1.out‚52910.12158203125 disk.sda1.out‚0 disk.sda.in‚0 cpu‚26.03700441354512 proc.cpu.threads‚17 network.sent‚ 2.0727456e+07 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚97.33445644378662* proc.memory.percent‚17.206907272338867ÿ‰&b4—)/ _wandbruntime‚11109  _runtime‚11109ï5ÚË̘)Æ ' lr-AdamW/pg1‚0.00013212341502885804 ' lr-AdamW/pg2‚0.00013212341502885804 trainer/global_step‚13049 % _timestamp‚1.7842839172802088e+09 _runtime‚11109.405816653 _step‚1072ø‘§fÑ™)Ë ' lr-AdamW/pg2‚0.00013212341502885804 trainer/global_step‚13049 % _timestamp‚1.7842839172802088e+09 _runtime‚11109.405816653 _step‚1072 ' lr-AdamW/pg1‚0.00013212341502885804°;CÑ4š)/ _wandbruntime‚11109  _runtime‚11109ÇJ£6ó›)€P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120869113 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224331887 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120859136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13799579567616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28088091 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450831692239872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13953132509184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224321908 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1983277410492416 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103462772 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120859136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1986855640077481 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1997076973020672 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1973076757557248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436878446249984 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1986855282488231 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13799579567616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1997077331399168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50947134 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚1395322737fù™7664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1973076774596608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450831578759168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1073 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚13049 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _runtime‚11109.410361203 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1997076973020672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50946859 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚436878464862208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1983277393453056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13799920906752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13778524930983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103462772 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1983277410492416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22858771 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576”pt (œ)…P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1983277393453056 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120869113 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22858768 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 $ _timestamp‚1.784283917281276e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13799920906752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13778524930983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103462772 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1983277410492416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22858771 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120869113 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224331887 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120859136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13799579567616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28088091 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450831692239872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13953132509184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224321908 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1983277410492416 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103462772 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120859136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1986855640077481 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1997076973020672 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1973076757557248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436878446249984 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1986855282488231 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13799579567616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1997077331399168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50947134 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13953227377664 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1973076774596608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450831578759168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1073 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚13049 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _runtime‚11109.410361203 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1997076973020672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50946859 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚436878464862208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1983277393453056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120878179 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚436911875008512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13800592194560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1973223285268480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224338592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28090440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120868188 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13800933540864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224338592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 trainer/global_step‚13049 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13954166387200 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22860513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224348585 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1973223268229120 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103470404 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103470404 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103470406 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224348585 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1983424651522048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13779876534707 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50950700 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚450866144383488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.784283917892627e+09 _step‚1074 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541åb£{Ÿ)N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1987003161803187 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103470406 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50950700 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚450866144383488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.784283917892627e+09 _step‚1074 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚11110.018437615 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13779535984705 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120878179 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50950956 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1987002804213825 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1997225585062912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚436911893620736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120878179 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚436911875008512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13800592194560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1973223285268480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224338592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28090440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120868188 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13800933540864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224338592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 trainer/global_step‚13049 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13954166387200 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/act {”Òšive_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22860513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224348585 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1973223268229120 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103470404 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103470404 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103470406 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224348585 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1983424651522048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13779876534707 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E ¤)‚(:‚ ’†èÒþïĤ% disk.nvme0c0n1.in‚144837.75390625# disk.nvme0n1.in‚144837.75390625& disk.nvme0n1.out‚52919.79833984375! disk.rbd1.out‚111.94970703125 disk.dm-0.in‚144837.75390625 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2819.1796875 disk.sda3.in‚0 network.recv‚ 2.976787e+06 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 network.sent‚ 2.0732526e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 434.78515625 disk.sda2.in‚0$ memory_percent‚97.33881950378418' proc.memory.availableMB‚ 436.0078125# disk.dm-0.out‚52919.79833984375 disk.dm-1.in‚0 disk.dm-1.out‚ 434.78515625 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.sda3.out‚ 434.78515625 proc.cpu.threads‚17) disk./.usagePercent‚36.82908284706921( disk.nvme0c0n1.out‚52919.79833984375* proc.memory.percent‚17.206907272338867 cpu‚26.036504632730445$ disk./.usageGB‚548.7799491882324 disk.rbd3.in‚01‚ë¿¥)‚(:’ ¡†èÒÈï¤ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ ™ ¦)‚(:€ ¡†èÒƒ€Ÿ¦ disk.sda.in‚0 disk.sda3.out‚ 435.03515625 network.recv‚ 2.981548e+06 disk.rbd3.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚52928.4326171875 disk.drbd2165.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚52928.4326171875 disk.sda2.in‚0" proc.memory.rssMB‚ 2819.1796875 disk.sda3.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.206907272338867 cpu‚26.036089444190175 disk.rbd3.in‚0# disk.nvme0n1.in‚144838.58203125 disk.rbd1.in‚0.0625$ memory_percent‚97.33586311340332 proc.cpu.threads‚17% disk.nvme0c0n1.in‚144838.58203125! disk.rbd1.out‚112.07470703125 network.sent‚ 2.0797662e+07* disk./.usagePercent‚36.829114335995925 disk.dm-0.in‚144838.58203125 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚548.7804183959961 disk.drbd2165.in‚0 disk.dm-1.out‚ 435.03515625% disk.nvme0n1.out‚52928.4326171875 disk.sda.out‚ 435.03515625 disk.sda2.out‚0' proc.memory.availableMB‚ 436.4921875ÒÖd’§)‚(:‡ °†èÒ“é÷¤ gpu.0.gpu‚30.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ѳÊþ¨)‚(:ó °†èÒÛ‰€§* proc.memory.percent‚17.206907272338867 network.sent‚ 2.0802735e+07# disk.nvme0c0n1.in‚ 144839.296875 network.recv‚ 2.984486e+06( disk.nvme0c0n1.out‚52939.61572265625" proc.memory.rssMB‚ 2819.1796875 disk.sda2.in‚0 disk.sda3.out‚ 435.28125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0# memory_percent‚97.3374605178833 cpu‚26.035679907663717 disk.rbd3.out‚0# disk./.usageGB‚548.780200958252 disk.drbd2165.in‚0 disk.sda.out‚ 435.28125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚52939.61572265625 disk.sda2.out‚0 disk.dm-1.out‚ 435.28125! disk.rbd1.out‚112.07470703125 disk.dm-0.in‚ 144839.296875# disk.dm-0.out‚52939.61572265625! disk.nvme0n1.in‚ 144839.296875 disk.drbd2165.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 436.23046875* disk./.usagePercent‚36.829099743566466 disk.rbd0.in‚0 disk.rbd1.in‚0.0625‚} R4©)/ _wandbruntime‚11149  _runtime‚11149^%£ ̪)Æ ' lr-AdamW/pg1‚0.00013042681154066055 ' lr-AdamW/pg2‚0.00013042681154066055 trainer/global_step‚13099 % _timestamp‚1.7842839572265024e+09 _runtime‚11149.352079462 _step‚1076Q¹Y.Ñ«)Ë ' lr-AdamW/pg2‚0.00013042681154066055 trainer/global_step‚13099 % _timestamp‚1.7842839572265024e+09 _runtime‚11149.352079462 _step‚1076 ' lr-AdamW/pg1‚0.00013042681154066055´ßL‡4¬)/ _wandbruntime‚11149  _runtime‚11149–³j!(­)P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2004490078382592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1994231432568739 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚13099 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1994231790157989 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2004489720004096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2004490078382592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51138766 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14004217454080 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13829481967781 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121323565 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225167939 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452554008859136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225157960 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1990639444934656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13850616408576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121323565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103844374 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1990639444934656 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103844372 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452554120242688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438549884176384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13850275069440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438O”%Ú (®)†P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452554008859136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225157960 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1990639444934656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13850616408576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121323565 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103844374 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1990639444934656 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103844372 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452554120242688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438549884176384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13850275069440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2004490078382592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1994231432568739 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚13099 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1994231790157989 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2004489720004096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2004490078382592 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51138766 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14004217454080 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13829481967781 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121323565 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225167939µ+^µƒ4¯)/ _wandbruntime‚11149  _runtime‚11149gÀÚk°)‹N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13851629042688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28194517 trainer/global_step‚13099 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842839578402646e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103852006 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13851287696384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13830493021615 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452588454626304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2004638332046336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51142280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end›ìÒ#¦/reserved_bytes.small_pool.peak‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1990786685964288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14005238649856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225184637 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51142543 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1990786703003648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28194777 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14005141691392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1980548818862080 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121332631 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103852004 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2004638332046336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13851287696384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225174644 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438583312934912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121322640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1994379311883695 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13830152471613 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51142280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1990786685964288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14005238649856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225184637 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51142543 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1990786703003648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28194777 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14005141691392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1980548818862080 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121332631 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103852004 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2004638332046336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13851287696384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225174644 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438583312934912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121322640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1994379311883695 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13830152471613 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169¶Pí$%4²)/ _wandbruntime‚11149  _runtime‚11149ªÌÁ³)Š 0loss/FRt2/assignment_loss‚0.8304721470993598 3loss/SRqqt2/assignment_loss‚0.42877154018929003 epoch‚32 2loss/FBt1/assignment_loss‚0.002662939964040463 1loss/SRqqt1/detection_loss‚0.8506042985590933 /loss/FBt1/detection_loss‚0.4661573484936276 $ _timestamp‚1.784283957843129e+09 3loss/SRqqt1/assignment_loss‚0.41946103209791885 /loss/FBt2/detection_loss‚0.4948126111800472 trainer/global_step‚13099 _runtime‚11149.968450953 _step‚1079 %loss/total_loss‚7.321838609474634 0loss/FRt1/assignment_loss‚1.5269824330438704 3loss/FBt2/assignment_loss‚0.0040287225990664885 0loss/FRt1/detection_loss‚0.47723587026088743 /loss/FRt2/detection_loss‚1.2024000388371867 1loss/SRqqt2/detection_loss‚0.6182496271502465Æ*ô) _runtime‚11149.968450953 _step‚1079 %loss/total_loss‚7.321838609474634 0loss/FRt1/assignment_loss‚1.5269824330438704 3loss/FBt2/assignment_loss‚0.0040287225990664885 0loss/FRt1/detection_loss‚0.47723587026088743 /loss/FRt2/detection_loss‚1.2024000388371867 1loss/SRqqt2/detection_loss‚0.6182496271502465 0loss/FRt2/assignment_loss‚0.8304721470993598 3loss/SRqqt2/assignment_loss‚0.42877154018929003 epoch‚32 2loss/FBt1/assignment_loss‚0.002662939964040463 1loss/SRqqt1/detection_loss‚0.8506042985590933 /loss/FBt1/detection_loss‚0.4661573484936276 $ _timestamp‚1.784283957843129e+09 3loss/SRqqt1/assignment_loss‚0.41946103209791885 /loss/FBt2/detection_loss‚0.4948126111800472 trainer/global_step‚13099·w¿±œµ)‚(:‘ ¿†èÒ©ƒê¢ gpu.0.gpu‚24.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%–¾ÿ¶)‚(:ô ¿†èÒ’ä disk.dm-1.out‚ 435.56640625 disk.sda1.in‚0! disk.nvme0n1.in‚ 144839.296875 proc.cpu.threads‚17 network.sent‚ 2.0805699e+07% disk.nvme0c0n1.out‚52943.66796875* proc.memory.percent‚17.206907272338867) disk./.usagePercent‚36.82911638405619 disk.sda.in‚0 disk.sda.out‚ 435.56640625# disk.nvme0n1.out‚52943.66796875( proc.memory.availableMB‚ 435.98046875" proc.memory.rssMB‚ 2819.1796875 cpu‚26.035264905934316 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 435.56640625 disk.dm-0.in‚ 144839.296875 disk.drbd2165.in‚0$ memory_percent‚97.33898639678955 disk.sda2.out‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚112.13720703125 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.out‚52943.66796875$ disk./.usageGB‚548.7804489135742 disk.rbd0.in‚0 disk.rbd3.in‚0 network.recv‚ 2.988085e+06# disk.nvme0c0n1.in‚ 144839.296875>€ßž·)‚(:“ ΆèÒóÖ㣠gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Ær û¸)‚(:ð ΆèÒöŸ×¤ disk.rbd0.in‚0 disk.dm-1.out‚ 435.9453125# disk.dm-0.out‚52952.38525390625 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 144840.125* proc.memory.percent‚17.206907272338867$ disk./.usageGB‚548.7804527282715 network.recv‚ 2.992153e+06) disk./.usagePercent‚36.82911664006373 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 144840.125 disk.sda3.out‚ 435.9453125! disk.rbd1.out‚112.13720703125 disk.nvme0n1.in‚ 144840.125 disk.sda1.in‚0 disk.sda.out‚ 435.9453125 disk.sda2.out‚0 disk.sda3.in‚0# memory_percent‚97.3376989364624" proc.memory.rssMB‚ 2819.1796875 network.sent‚ 2.0868733e+07 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 436.19140625 proc.cpu.threads‚17& disk.nvme0n1.out‚52952.38525390625( disk.nvme0c0n1.out‚52952.38525390625 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 cpu‚26.034838216156995u†4¹)/ _wandbruntime‚11189  _runtime‚11189 L –̺)Æ _runtime‚11189.327611483 _step‚1080 ' lr-AdamW/pg1‚0.00012873701028014395 ' lr-AdamW/pg2‚0.00012873701028014395 trainer/global_step‚13149 % _timestamp‚1.7842839972020109e+09)«{­Ñ»)Ë _step‚1080 ' lr-AdamW/pg1‚0.00012873701028014395 ' lr-AdamW/pg2‚0.00012873701028014395 trainer/global_step‚13149 % _timestamp‚1.7842839972020109e+09 _runtime‚11189.327611483¸(·$Z4¼)/ _wandbruntime‚11189  _runtime‚11189}xP%½)P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2011902466987520 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121768040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51329906 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28296635 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1998001496416256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2001607940238497 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1998001513455616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104225974 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454276403414528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23033268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2011902466987520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13901311910400 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1998001496416256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121768040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚11189.328812286 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226003991 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2011902825366016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13880098454689 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13879757904799 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104225972 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13900970571264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14055182471680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104225974 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104225972 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121778017 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1987727841783808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 $ _timestamp‚1.784283997203053e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13900970571264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13901311910400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440221340715008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225994012 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51329627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121778017 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1987727824744448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454276523186688 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14055081311744 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2001607582649247 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2011902466987520 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121768040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51329906 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28296635 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1998001496416256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2001607940238497 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1998001513455616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104225974 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454276403414528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23033268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2011902466987520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13901311910400 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1998001496416256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121768040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚11189.328812286 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226003991 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2011902825366016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13880098454689 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13879757904799 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104225972 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13900970571264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14055182471680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104225974 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104225972 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121778017 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1987727841783808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 $ _timestamp‚1.784283997203053e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13900970571264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13901311910400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440221340715008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225994012 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51329627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121778017 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1987727824744448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454276523186688 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14055081311744 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2001607582649247 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226020689 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1987874335416320 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13902324544512 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1998148737445888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121777092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23035013 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226020689 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13902324544512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440254769473536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440254750861312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28298395 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1998148754485248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2001755104374841 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104233604 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454310851227648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104233606 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226010696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51333682 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13880768958521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14056193130496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784283997815357e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13901983198208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28298666 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13881109508523 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104233604 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51333408 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976Îｊ'Á)N QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1998148737445888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121777092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23035013 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226020689 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13902324544512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440254769473536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440254750861312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28298395 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1998148754485248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2001755104374841 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104233604 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454310851227648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104233606 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226010696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51333682 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13880768958521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14056193130496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784283997815357e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13901983198208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28298666 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13881109508523 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104233604 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51333408 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226010696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚13149 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1998148737445888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2012051079029760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226020689 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1987874335416320 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13902324544512 EVÅ)‚(:“ ݆èÒô³Ï¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllo¨’Ã6Hcated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.842600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_›X Æ)‚(:þ ݆èÒ²·½¥ disk.rbd0.out‚0& disk.nvme0n1.out‚52971.30224609375 disk.sda.in‚0! disk.rbd1.out‚112.19970703125 disk.rbd3.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 435.9609375% disk.nvme0c0n1.in‚144840.83984375( disk.nvme0c0n1.out‚52971.30224609375 disk.sda2.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚52971.30224609375* proc.memory.percent‚17.206907272338867 cpu‚26.034409627413822 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 438.0703125 network.sent‚ 2.0886824e+07 network.recv‚ 2.995451e+06) disk./.usagePercent‚36.82942948127057 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 438.0703125 disk.rbd1.in‚0.0625 disk.dm-0.in‚144840.83984375# disk.nvme0n1.in‚144840.83984375 disk.drbd2165.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2819.1796875 proc.cpu.threads‚17$ disk./.usageGB‚548.7851142883301 disk.sda.out‚ 438.0703125# memory_percent‚97.3391056060791w“ѓÇ)‚(:ˆ ì†èÒÉ„Ù¢ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚238.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.6244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûõ4¤ È)‚(: ì†èÒÝÖº¥* proc.memory.percent‚17.206907272338867 network.recv‚ 2.999908e+06 disk.sda1.in‚0( disk.nvme0c0n1.out‚52988.00439453125 disk.dm-1.out‚ 440.35546875! disk.rbd1.out‚112.19970703125* disk./.usagePercent‚36.829207522738386 disk.drbd2165.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 436.23046875$ disk./.usageGB‚548.7818069458008# disk.nvme0n1.in‚144841.66796875% disk.nvme0c0n1.in‚144841.66796875 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 cpu‚26.03398609261297 disk.sda1.out‚0 disk.sda3.out‚ 440.35546875 disk.sda2.out‚0" proc.memory.rssMB‚ 2819.1796875& disk.nvme0n1.out‚52988.00439453125 disk.sda.out‚ 440.35546875 disk.dm-1.in‚0 disk.rbd3.out‚0# memory_percent‚97.3374605178833 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚52988.00439453125 proc.cpu.threads‚17 network.sent‚ 2.093696e+07 disk.dm-0.in‚144841.66796875¡1j“É)‚(:ˆ û†èÒ¯¦¤ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚248.333' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ÞCˆ Ê)‚(: û†èÒ‘Æ…¦( disk.nvme0c0n1.out‚52991.47119140625 disk.sda.out‚ 441.37890625 disk.rbd1.in‚0.0625! disk.rbd1.out‚112.26220703125" proc.memory.rssMB‚ 2819.1796875 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚144841.66796875 disk.rbd0.out‚0& disk.nvme0n1.out‚52991.47119140625 disk.sda1.out‚0$ memory_percent‚97.33591079711914 cpu‚26.033582960736517$ disk./.usageGB‚548.7797966003418 disk.dm-1.out‚ 441.37890625# disk.nvme0n1.in‚144841.66796875& proc.memory.availableMB‚ 436.484375 network.sent‚ 2.0942091e+07) disk./.usagePercent‚36.82907260676784 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 441.37890625 disk.rbd0.in‚0% disk.nvme0c0n1.in‚144841.66796875 network.recv‚ 3.003553e+06 disk.rbd3.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.206907272338867# disk.dm-0.out‚52991.47119140625âœ14Ë)/ _wandbruntime‚11229  _runtime‚11229Wã=øÌÌ)Æ ' lr-AdamW/pg1‚0.00012705411978008592 ' lr-AdamW/pg2‚0.00012705411978008592 trainer/global_step‚13199 % _timestamp‚1.7842840369194193e+09 _runtime‚11229.045014067 _step‚1084ÞÜiãÑÍ)Ë ' lr-AdamW/pg1‚0.00012705411978008592 ' lr-AdamW/pg2‚0.00012705411978008592 trainer/global_step‚13199 % _timestamp‚1.7842840369194193e+09 _runtime‚11229.045014067 _step‚1084¼ÀÓ»e4Î)/ _wandbruntime‚11229  _runtime‚11229äV%Ê(Ï)P ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2008984090319005 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2008983732729755 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13951666073088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104607572 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13930714941597 _step‚1085 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1995053375377408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441892760029184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122222492 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2019315213970944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226840043 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51520110 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455998801862656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104607574 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2005363547897856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226830064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2019315213970944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2019315572349440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚11229.046259229 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104607572 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28399324 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104607574 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51519842 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122222492 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14106041833472 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2005363564937216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14106128313344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚13199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13952007412224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122232469 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23120518 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13930374391707 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13951666073088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Ìq (Ð)†P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104607574 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51519842 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122222492 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14106041833472 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2005363564937216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14106128313344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚13199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13952007412224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122232469 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23120518 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13930374391707 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13951666073088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2008984090319005 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2008983732729755 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13951666073088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104607572 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13930714941597 _step‚1085 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1995053375377408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441892760029184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122222492 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2019315213970944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226840043 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51520110 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455998801862656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104607574 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2005363547897856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226830064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2019315213970944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2019315572349440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚11229.046259229 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104607572 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28399324 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0½Ü&Œ4Ñ)/ _wandbruntime‚11229  _runtime‚11229EIè4ÃÒ)ŠN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122241535 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2019463826013184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1995199886049280 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226846748 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51523657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14107157619200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚13199 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2019463826013184 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23122266 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104615206 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13953020046336 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842840375314846e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441926207399936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14107081632256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2005510788927488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226846748 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚456033270419968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13931385445429 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2005510805966848 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13953020046336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441926188787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2005510805966848 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226856741 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13931725995431 H@DeviceStatsMonitor.on_train_batch_end/segme%Sù)M nt.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856741 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚456033365019136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104615206 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104615204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23122263 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1995199869009920 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2005510788927488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13952678700032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13952678700032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28401652 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2009131612044711 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚11229.657333571 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28401394 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2009131254455349_{p‡'Ó)N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226856741 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13931725995431 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856741 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚456033365019136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104615206 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104615204 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23122263 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1995199869009920 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2005510788927488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13952678700032 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13952678700032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28401652 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2009131612044711 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚11229.657333571 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28401394 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2009131254455349 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122241535 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2019463826013184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1995199886049280 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226846748 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51523657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14107157619200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚13199 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2019463826013184 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23122266 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104615206 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13953020046336 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842840375314846e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441926207399936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14107081632256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2005510788927488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226846748 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚456033270419968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13931385445429 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2005510805966848 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13953020046336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441926188787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2005510805966848 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0¾°Ø4Ô)/ _wandbruntime‚11229  _runtime‚11229ã&µ“Õ)Œ 3loss/SRqqt1/assignment_loss‚0.36502684005918085 0loss/FBt1/detection_loss‚0.34483503901117274 trainer/global_step‚13199 0loss/FRt1/assignment_loss‚1.5111058604395067 2loss/FBt2/assignment_loss‚0.004955089584611521 1loss/SRqqt2/detection_loss‚0.5710109968052469 0loss/FBt2/detection_loss‚0.46818828268817836 epoch‚32 3loss/SRqqt2/assignment_loss‚0.48135000645538256 3loss/FBt1/assignment_loss‚0.0006458821394355052 /loss/FRt1/detection_loss‚0.4946463767193278 %loss/total_loss‚7.157016163875944 _step‚1087 /loss/FRt2/detection_loss‚1.2949585353035051 1loss/SRqqt1/detection_loss‚0.7841178128194861 % _timestamp‚1.7842840375349312e+09 _runtime‚11229.660272231 0loss/FRt2/assignment_loss‚0.8361754418509098ÌÖ)‘ 0loss/FRt2/assignment_loss‚0.8361754418509098 3loss/SRqqt1/assignment_loss‚0.36502684005918085 0loss/FBt1/detection_loss‚0.34483503901117274 trainer/global_step‚13199 0loss/FRt1/assignment_loss‚1.5111058604395067 2loss/FBt2/assignment_loss‚0.004955089584611521 1loss/SRqqt2/detection_loss‚0.5710109968052469 0loss/FBt2/detection_loss‚0.46818828268817836 epoch‚32 3loss/SRqqt2/assignment_loss‚0.48135000645538256 3loss/FBt1/assignment_loss‚0.0006458821394355052 /loss/FRt1/detection_loss‚0.4946463767193278 %loss/total_loss‚7.157016163875944 _step‚1087 /loss/FRt2/detection_loss‚1.2949585353035051 1loss/SRqqt1/detection_loss‚0.7841178128194861 % _timestamp‚1.7842840375349312e+09 _runtime‚11229.660272231¿„l븑×)‚(:† Š‡èÒŸÐÊ¢ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ`ìÅØ)‚(:÷ Š‡èÒþ°È¥ disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0# memory_percent‚97.3391056060791" proc.memory.rssMB‚ 2819.1796875 disk.sda3.out‚ 441.6953125 disk.sda1.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 441.6953125 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚36.82907286277538 disk.sda.in‚0 disk.drbd2165.in‚0 cpu‚26.034870260594726$ disk.nvme0c0n1.in‚144842.3828125' disk.nvme0c0n1.out‚53001.1220703125 disk.sda2.out‚0 disk.rbd1.in‚0.0625' proc.memory.availableMB‚ 435.9609375! disk.rbd1.out‚112.26220703125" disk.dm-0.out‚53001.1220703125" disk.nvme0n1.in‚144842.3828125% disk.nvme0n1.out‚53001.1220703125 disk.rbd0.in‚0 network.recv‚ 3.00699e+06 disk.sda.out‚ 441.6953125 disk.dm-0.in‚144842.3828125 disk.rbd0.out‚0 network.sent‚ 2.0960192e+07$ disk./.usageGB‚548.7798004150391* proc.memory.percent‚17.206907272338867¸me“4Ù)/ _wandbruntime‚11246  _runtime‚11246–Kc„n=Ú)èz @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36267778277397156 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7810042500495911 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 &particle/f_score‚0.634846031665802 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6validation_average_jet_accuracy‚0.3973931670188904 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7241390943527222 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998185396194458 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27332714200019836 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19623111188411713 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24894359707832336 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_3_of_4‚0.0829051211476326 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26521292328834534 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5450348258018494 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0495605543255806 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4979040026664734 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8708138465881348 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 _step‚1088 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3137586712837219 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3573635518550873 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2447589933872223 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7370455861091614 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2537204325199127 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954172968864441 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3140649199485779 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.06540894508361816 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚32 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6468014717102051 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7209834456443787 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22949817776679993 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05771271511912346 particle/accuracy_2_of_2‚1 +particle/sensitivity‚0.6276200413703918 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022985048592090607 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888571500778198 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8449629545211792 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 % _timestamp‚1.7842840547979705e+09 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7025923132896423 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3044811487197876 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6229966282844543 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6826869249343872 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.023198746144771576 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6920663118362427 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1341666728258133 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987881183624268 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970333933830261 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7744865417480469 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8129251599311829 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6485673785209656 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.620655357837677 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6777470111846924 *jet/accuracy_2_of_4‚0.8153958320617676 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9320101737976074 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9258241057395935 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6591389179229736 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25746166706085205 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8601140379905701 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 particle/accuracy_4_of_5‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.275249183177948 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6986481547355652 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2630152106285095 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21012133359909058 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9966940879821777 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09958887100219727 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 )jet/accuracy_1_of_1‚0.358951210975647 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6421046853065491 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23781156539916992 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7841858863830566 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962462186813354 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚11246.924262463 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884147644042969 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958134651184 jet/accuracy_1_of_5‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8124528527259827 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *jet/accuracy_1_of_4‚0.9787419438362122 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19878803193569183 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997230052947998 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 trainer/global_step‚13199 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6711453199386597 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960261583328247 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7396636009216309 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7003999948501587 particle/accuracy_1_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8219766020774841 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33012279868125916 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.568863570690155 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6310634016990662 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5904756784439087 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3858696520328522 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3045397400856018 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8001495599746704 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 (jet/accuracy_1_of_3‚0.93434077501297 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3044811487197876 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6957012414932251 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7770975232124329 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6289254426956177 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7410687804222107 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3022613823413849 jet/accuracy_3_of_5‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020133361220359802 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6269187927246094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8708138465881348 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.558914065361023 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022289464250206947 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27701640129089355 *jet/accuracy_1_of_2‚0.8071686029434204 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8701192736625671 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2957349419593811 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.384653776884079 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7325627207756042 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03996298462152481 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_4_of_4‚0.011098379269242287 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6657708287239075 particle/accuracy_2_of_3‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020133361220359802 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5692290663719177 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9927580952644348 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49286720156669617 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02244255505502224 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26439738273620605 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6691316366195679 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42200398445129395 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26277774572372437 +jet/accuracy_2_of_2‚0.16609779000282288 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083432674408 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7909108400344849 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9642454981803894 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6073333024978638 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3160591125488281 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6491233110427856 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47046130895614624 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.052508581429719925 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1096726506948471 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3144667446613312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6491233110427856 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7859612703323364 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 )loss/val_total_loss‚7.897631667745086 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6405807137489319 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5524721145629883 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1848912537097931 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7944846153259277 +particle/specificity‚0.9044014811515808 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5270832777023315 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7029616832733154 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9831644892692566 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993018627166748 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6619942784309387 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375èQÆÛ)íz 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3137586712837219 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3573635518550873 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2447589933872223 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7370455861091614 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2537204325199127 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954172968864441 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3140649199485779 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.06540894508361816 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚32 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6468014717102051 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7209834456443787 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.229498177766799 ‡Ý½­893 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05771271511912346 particle/accuracy_2_of_2‚1 +particle/sensitivity‚0.6276200413703918 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022985048592090607 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888571500778198 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5555555820465088 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8449629545211792 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 % _timestamp‚1.7842840547979705e+09 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7025923132896423 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3044811487197876 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6229966282844543 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6826869249343872 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.023198746144771576 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6920663118362427 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1341666728258133 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987881183624268 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970333933830261 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7744865417480469 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8129251599311829 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6485673785209656 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.620655357837677 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6777470111846924 *jet/accuracy_2_of_4‚0.8153958320617676 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9320101737976074 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9258241057395935 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6591389179229736 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25746166706085205 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8601140379905701 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 particle/accuracy_4_of_5‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.275249183177948 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6986481547355652 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2630152106285095 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21012133359909058 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9966940879821777 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09958887100219727 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 )jet/accuracy_1_of_1‚0.358951210975647 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6421046853065491 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23781156539916992 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7841858863830566 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962462186813354 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚11246.924262463 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884147644042969 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958134651184 jet/accuracy_1_of_5‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8124528527259827 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *jet/accuracy_1_of_4‚0.9787419438362122 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19878803193569183 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997230052947998 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 trainer/global_step‚13199 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6711453199386597 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960261583328247 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7396636009216309 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7003999948501587 particle/accuracy_1_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8219766020774841 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33012279868125916 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.568863570690155 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6310634016990662 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5904756784439087 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3858696520328522 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3045397400856018 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8001495599746704 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 (jet/accuracy_1_of_3‚0.93434077501297 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3044811487197876 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6957012414932251 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7770975232124329 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6289254426956177 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7410687804222107 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3022613823413849 jet/accuracy_3_of_5‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020133361220359802 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6269187927246094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8708138465881348 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.558914065361023 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022289464250206947 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27701640129089355 *jet/accuracy_1_of_2‚0.8071686029434204 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8701192736625671 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2957349419593811 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.384653776884079 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7325627207756042 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03996298462152481 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_4_of_4‚0.011098379269242287 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6657708287239075 particle/accuracy_2_of_3‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020133361220359802 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5692290663719177 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9927580952644348 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49286720156669617 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02244255505502224 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26439738273620605 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6691316366195679 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.42200398445129395 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.550000011920929 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26277774572372437 +jet/accuracy_2_of_2‚0.16609779000282288 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3177083432674408 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7909108400344849 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9642454981803894 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6073333024978638 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3160591125488281 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6491233110427856 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47046130895614624 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.052508581429719925 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1096726506948471 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3144667446613312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6491233110427856 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7859612703323364 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 )loss/val_total_loss‚7.897631667745086 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6405807137489319 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5524721145629883 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1848912537097931 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7944846153259277 +particle/specificity‚0.9044014811515808 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5270832777023315 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7029616832733154 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9831644892692566 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993018627166748 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6619942784309387 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36267778277397156 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7810042500495911 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 &particle/f_score‚0.634846031665802 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6validation_average_jet_accuracy‚0.3973931670188904 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7241390943527222 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998185396194458 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27332714200019836 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19623111188411713 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24894359707832336 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_3_of_4‚0.0829051211476326 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26521292328834534 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5450348258018494 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0495605543255806 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4979040026664734 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8708138465881348 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 _step‚1088 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059À(Ö[–žÜ)‚(:“ ™‡èÒߟ¢ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Ãé~ Ý)‚(:ÿ ™‡èÒè©®¥ disk.dm-1.in‚0 disk.rbd1.in‚0.0625( disk.nvme0c0n1.out‚53008.08642578125 disk.sda.out‚ 441.98828125 disk.rbd0.in‚0" disk.nvme0n1.in‚144843.2109375& disk.nvme0n1.out‚53008.08642578125 disk.sda.in‚0# disk.dm-0.out‚53008.08642578125( proc.memory.availableMB‚ 434.43359375 network.sent‚ 2.1010203e+07$ disk.nvme0c0n1.in‚144843.2109375 disk.dm-1.out‚ 441.98828125$ memory_percent‚97.34842777252197) proc.memory.percent‚17.21494197845459 network.recv‚ 3.011508e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0! disk.rbd1.out‚112.38720703125# proc.memory.rssMB‚ 2820.49609375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚144843.2109375 cpu‚26.03869820865001) disk./.usagePercent‚36.82906083042127 disk.sda3.out‚ 441.98828125 disk.rbd3.in‚0 disk.rbd3.out‚0$ disk./.usageGB‚548.7796211242676 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.in‚0'÷MeÞ)‚(:’ ¨‡èÒÝ€¤ gpu.0.gpu‚50.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B~ ß)‚(:€ ¨‡èÒ•©õ¥# disk.nvme0n1.in‚144843.92578125 disk.dm-1.in‚0$ memory_percent‚97.34811782836914 network.sent‚ 2.1067471e+07 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 442.41796875) proc.memory.percent‚17.21494197845459 cpu‚26.03826847684936 network.recv‚ 3.015483e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚144843.92578125) disk./.usagePercent‚36.82905417422538& disk.nvme0n1.out‚53016.27392578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144843.92578125 disk.sda.out‚ 442.41796875 disk.sda2.in‚0 disk.sda.in‚0! disk.rbd1.out‚112.38720703125# disk.dm-0.out‚53016.27392578125# proc.memory.rssMB‚ 2820.49609375 proc.cpu.threads‚17$ disk./.usageGB‚548.7795219421387( disk.nvme0c0n1.out‚53016.27392578125 disk.sda3.in‚0 disk.dm-1.out‚ 442.41796875& proc.memory.availableMB‚ 434.484375ѧ–‘à)‚(:† ·‡èÒþÄ΢ gpu.0.gpu‚6.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙöÙ á)‚(: ·‡èÒ«¬ö¤ disk.rbd1.in‚0.0625 disk.sda.in‚0( disk./.usagePercent‚36.8291179201014 disk.dm-1.out‚ 442.84765625 disk.sda1.in‚0( disk.nvme0c0n1.out‚53023.31396484375 disk.sda3.out‚ 442.84765625 disk.dm-1.in‚0% disk.nvme0c0n1.in‚144844.75390625 cpu‚26.037869304201934$ disk./.usageGB‚548.7804718017578 disk.rbd3.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚ 3.019191e+06 disk.sda2.out‚0# proc.memory.rssMB‚ 2820.49609375 disk.sda1.out‚0) proc.memory.percent‚17.21494197845459 disk.sda.out‚ 442.84765625! disk.rbd1.out‚112.44970703125 disk.rbd3.in‚0 disk.sda3.in‚0 disk.dm-0.in‚144844.75390625& disk.nvme0n1.out‚53023.31396484375 disk.rbd0.out‚0# disk.dm-0.out‚53023.31396484375# disk.nvme0n1.in‚144844.75390625 disk.drbd2165.in‚0 disk.rbd0.in‚0$ memory_percent‚97.34814167022705( proc.memory.availableMB‚ 434.48046875 network.sent‚ 2.107274e+07 disk.sda2.in‚0^†4â)/ _wandbruntime‚11286  _runtime‚11286 ñ¸Ëã)Å ' lr-AdamW/pg1‚0.00012537824812939837 ' lr-AdamW/pg2‚0.00012537824812939837 trainer/global_step‚13249 $ _timestamp‚1.784284093924374e+09 _runtime‚11286.051387735 _step‚1089üŸ¾Ðä)Ê _step‚1089 ' lr-AdamW/pg1‚0.00012537824812939837 ' lr-AdamW/pg2‚0.00012537824812939837 trainer/global_step‚13249 $ _timestamp‚1.784284093924374e+09 _runtime‚11286.051387735ÁªÔu4å)/ _wandbruntime‚11286  _runtime‚11286˜¯°*(æ)P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105109838 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457755519879168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13999051153734 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51743222 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2014655166029824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122747266 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122747266 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2028675260810240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2028675619188736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51743482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443580128667648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443580110055424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13998710603844 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2014655166029824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14020436119552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2004287572185088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105109840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2018286623338822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28533191 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14175494206464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚13249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122757243 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2028675260810240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28532934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23210288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105109840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2014655183069184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227867083 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2014655183069184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14020094780416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2004287555145728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14020436119552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23210291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14175409823744 _step‚10906”i ç)†P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2018286265749572 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 227857104 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2028675619188736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14175494206464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚13249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122757243 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2028675260810240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28532934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23210288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105109840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2014655183069184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227867083 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.aÑà llocated‚2014655183069184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14020094780416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2004287555145728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14020436119552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23210291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14175409823744 _step‚1090 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105109838 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457755519879168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13999051153734 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51743222 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2014655166029824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122747266 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122747266 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2028675260810240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2028675619188736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51743482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443580128667648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443580110055424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13998710603844 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2014655166029824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14020436119552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2004287572185088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105109840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2018286623338822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28533191 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227873788 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227873788 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14176517491200 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105117472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7842840945349562e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14021448753664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2018433787475166 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14021107407360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2014802424098816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122766309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚443613557426176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105117472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 trainer/global_step‚13249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105117470 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28535259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122756318 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122756318 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28534988 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1091 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2018434145064528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51747295 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122766309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23212036 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2028823872852480 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443613538813952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2028823514466816 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2014802407059456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14176426824192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51747021 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14021448753664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227883781 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2004434065817600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14021107407360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105117470 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227883781 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14000062207568 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2028823872852480 ë'ê)N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14021107407360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2014802424098816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122766309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚443613557426176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105117472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 trainer/global_step‚13249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105117470 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28535259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122756318 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122756318 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28534988 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1091 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2018434145064528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51747295 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122766309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23212036 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2028823872852480 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443613538813952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2028823514466816 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2014802407059456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14176426824192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51747021 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14021448753664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227883781 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2004434065817600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14021107407360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105117470 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227883781 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14000062207568 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2028823872852480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227873788 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227873788 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14176517491200 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105117472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7842840945349562e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14021448753664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2018433787475166 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ÃÞt"«4ë)/ _wandbruntime‚11286  _runtime‚11286Éòü-ì)‰ epoch‚33 _step‚1092 0loss/FRt1/assignment_loss‚1.5512220691036245 3loss/FBt2/assignment_loss‚0.0011172376990552988 1loss/SRqqt1/detection_loss‚0.8606030577550956 % _timestamp‚1.7842840945385926e+09 _runtime‚11286.663927767 0loss/FRt2/assignment_loss‚0.8296101866649981 0loss/SRqqt2/detection_loss‚0.596860864763065 2loss/SRqqt1/assignment_loss‚0.3562902862024593 3loss/FBt1/assignment_loss‚0.0005096213441522352 /loss/FRt2/detection_loss‚1.0841261454358473 /loss/FBt1/detection_loss‚0.3838412128012734 /loss/FBt2/detection_loss‚0.5426154959797194 &loss/total_loss‚7.1294277295175466 trainer/global_step‚13249 2loss/SRqqt2/assignment_loss‚0.4055941241623462 /loss/FRt1/detection_loss‚0.5170374276059095YHí)Ž 3loss/FBt2/assignment_loss‚0.0011172376990552988 1loss/SRqqt1/detection_loss‚0.8606030577550956 % _timestamp‚1.7842840945385926e+09 _runtime‚11286.663927767 0loss/FRt2/assignment_loss‚0.8296101866649981 0loss/SRqqt2/detection_loss‚0.596860864763065 2loss/SRqqt1/assignment_loss‚0.3562902862024593 3loss/FBt1/assignment_loss‚0.0005096213441522352 /loss/FRt2/detection_loss‚1.0841261454358473 /loss/FBt1/detection_loss‚0.3838412128012734 /loss/FBt2/detection_loss‚0.5426154959797194 &loss/total_loss‚7.1294277295175466 trainer/global_step‚13249 2loss/SRqqt2/assignment_loss‚0.4055941241623462 /loss/FRt1/detection_loss‚0.5170374276059095 epoch‚33 _step‚1092 0loss/FRt1/assignment_loss‚1.5512220691036245Ä`1Œ›î)‚(: ƇèÒ—Ýü£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.921' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.38419999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 @ˆï)‚(:õ ƇèÒá–ú¤# disk./.usageGB‚548.780876159668$ disk.nvme0c0n1.out‚ 53027.5703125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.34823703765869 disk.sda3.out‚ 443.04296875" disk.nvme0n1.out‚ 53027.5703125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚144844.75390625 disk.rbd0.in‚0( proc.memory.availableMB‚ 434.46484375 disk.sda.out‚ 443.04296875 disk.dm-0.in‚144844.75390625) disk./.usagePercent‚36.82914505690003! disk.rbd1.out‚112.44970703125 disk.sda1.in‚0% disk.nvme0c0n1.in‚144844.75390625 disk.sda.in‚0 disk.drbd2165.in‚0 cpu‚26.03746635035865 disk.rbd3.in‚0 disk.dm-0.out‚ 53027.5703125 disk.sda1.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2820.49609375 disk.sda2.out‚0 disk.dm-1.out‚ 443.04296875) proc.memory.percent‚17.21494197845459 network.sent‚ 2.1077765e+07 network.recv‚ 3.022361e+06pþÒ“ð)‚(:ˆ Õ‡èÒ°þТ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë¤Ûúñ)‚(:ï Õ‡èÒÏûÄ¥* disk./.usagePercent‚36.829145824922634' disk.nvme0c0n1.out‚53035.3291015625 disk.rbd1.in‚0.0625 disk.sda3.out‚ 443.296875" disk.dm-0.out‚53035.3291015625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 144845.46875% disk.nvme0n1.out‚53035.3291015625 network.sent‚ 2.1140923e+07 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0! disk.rbd1.out‚112.57470703125 disk.rbd3.in‚0 disk.nvme0n1.in‚ 144845.46875$ memory_percent‚97.34978675842285 network.recv‚ 3.02691e+06$ disk./.usageGB‚548.7808876037598 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 144845.46875 disk.dm-1.out‚ 443.296875) proc.memory.percent‚17.21494197845459 proc.cpu.threads‚17 disk.dm-1.in‚0' proc.memory.availableMB‚ 434.2109375# proc.memory.rssMB‚ 2820.49609375 cpu‚26.03703987613419 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda.out‚ 443.296875*Òj²“ò)‚(:ˆ ä‡èÒú¤¤ gpu.0.gpu‚87.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 q½ß-ó)‚(:û ä‡èÒ컄¥ network.sent‚ 2.1145941e+07 disk.sda.out‚ 444.33203125 disk.dm-0.in‚ 144846.296875' proc.memory.availableMB‚ 434.1953125# proc.memory.rssMB‚ 2820.49609375 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚53049.27001953125$ memory_percent‚97.34988212585449& disk.nvme0n1.out‚53049.27001953125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 144846.296875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 proc.cpu.thFÜdÙreads‚17) disk./.usagePercent‚36.82901372503496$ disk./.usageGB‚548.7789192199707 disk.sda.in‚0 disk.sda3.in‚0 cpu‚26.03657559346434 network.recv‚ 3.030487e+06 disk.sda3.out‚ 444.33203125 disk.dm-1.out‚ 444.33203125 disk.rbd1.in‚0.0625! disk.nvme0n1.in‚ 144846.296875) proc.memory.percent‚17.21494197845459( disk.nvme0c0n1.out‚53049.27001953125 disk.sda2.in‚0! disk.rbd1.out‚112.57470703125 disk.rbd0.out‚0rråW4ô)/ _wandbruntime‚11325  _runtime‚11325þ‡ëÉõ)à trainer/global_step‚13299 $ _timestamp‚1.784284133855944e+09 _runtime‚11325.982367638 _step‚1093 & lr-AdamW/pg1‚0.0001237095029661857 & lr-AdamW/pg2‚0.0001237095029661857’¼F Îö)È $ _timestamp‚1.784284133855944e+09 _runtime‚11325.982367638 _step‚1093 & lr-AdamW/pg1‚0.0001237095029661857 & lr-AdamW/pg2‚0.0001237095029661857 trainer/global_step‚13299ņ`×Ò4÷)/ _wandbruntime‚11325  _runtime‚11325…¾(ø)€P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14049327090752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2036088366172160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784284133856735e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228703135 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123201718 trainer/global_step‚13299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14070790282240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14070790282240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚445251547981824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14226780751360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123211695 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2011613088739328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105491438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123201718 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2022017217511424 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2036088366172160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2036088007793664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2025662415830080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2022017234550784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51935012 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28637203 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105491440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2022017234550784 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2022017217511424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2025662773419330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2011613105778688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14226685882880 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123211695 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14071131621376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228693156 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23297541 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976»e»Ä (ù)…P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105491438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2036088007793664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2025662415830080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2022017234550784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51935012 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28637203 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105491440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2022017234550784 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2022017217511424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2025662773419330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2011613105778688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14226685882880 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123211695 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14071131621376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228693156 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23297541 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14049327090752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2036088366172160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784284133856735e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228703135 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123201718 trainer/global_step‚13299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14070790282240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14070790282240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚445251547981824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14226780751360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123211695 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2011613088739328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105491438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123201718 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2022017217511424 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2036088366172160 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14050338144474 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459512685792256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14227799718912 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228719833 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105499072 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2025809937555674 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14072144255488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123220761 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2022164475580416 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2022164458541056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14072144255488 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14071802909184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123210770 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51938581 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 % _timestamp‚1.7842841344740598e+09}VÉŒü)N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23299283 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2036236619835904 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2036236261450240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28639298 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2025810295145036 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14071802909184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983_Š"3 _step‚1095 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228709840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51938581 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 % _timestamp‚1.7842841344740598e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51938831 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23299286 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2011759616450560 _runtime‚11326.599906647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123220761 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2036236619835904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚445284976740352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14050338144474 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459512685792256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14227799718912 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228719833 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105499072 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2025809937555674 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14072144255488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123220761 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2022164475580416 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2022164458541056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14072144255488 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14071802909184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123210770 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14121827123200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14121827123200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2043500754777088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105873038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229529208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14278001875968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2043500754777088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2043501113155584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28741549 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2018938622332928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52126056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123666147 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123656170 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28741268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0"©nO€!‰*†P RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14278092550144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14099943577660 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7842841736091633e+09 _step‚1098 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123656170 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123666147 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2033038565910588 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2018938639372288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105873040 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2043500754777088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105873038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229529208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14278001875968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2043500754777088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2043501113155584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28741549 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2018938622332928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52126056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123666147 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123656170 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28741268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2029379286032384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446922985908224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2029379268993024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23384791 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461200987784192 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2033038923499838 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446923004520448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229539187 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14121827123200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14121827123200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105880670 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105880672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52130140 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23386533 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2033186087636182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚13349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229545892 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2043649366819328 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52129875 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2033186445225544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23386536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2019085133004800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14122498411008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14279029485056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446956433278976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229555885 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123665222 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2029526510022656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚11366.345986773 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461235444151808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438bâŠ'Œ*N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2029526510022656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚13349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229545892 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2043649366819328 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52129875 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2033186445225544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23386536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2019085133004800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14122498411008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14279029485056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446956433278976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229555885 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123665222 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2029526510022656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚11366.345986773 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461235444151808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229545892 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229555885 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14122498411008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105880670 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105880672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52130140 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23386533 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2033186087636182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2050913501760512 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106254638 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448594423834624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124120599 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2036741320474624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2026264155926528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2050913501760512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124110622 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462923727984128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1102 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124110622 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14329388532224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2026264172965888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14150900614458 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124120599 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2050913860139008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52316769 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2050913860139008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106254640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23472041 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14329304149504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106254640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28844984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0µGþŠ (›*†P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28844731 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842842135476553e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23472038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2040415073580346 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230375239 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2050913501760512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124110622 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462923727984128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1102 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124110622 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14329388532224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2026264172965888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14150900614458 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124120599 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2050913860139008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52316769 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2050913860139008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106254640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23472041 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14329304149504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106254640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28844984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 _runtime‚11405.673431071 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14172181285888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106254638 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52317025 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2040414715991096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230365260 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14172522625024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2050913501760512 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106254638 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448594423834624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124120599 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2036741320474624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2026264155926528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230391937 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124119674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230381944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124129665 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2036888578543616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14173193912832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14330337533952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106262272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2051062113802752 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2036888578543616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2036888561504256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14151911668292 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14173193912832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚13399 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230391937 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2040562237716690 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2051061755417088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2036888561504256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106262270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52320603 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1103 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14173535259136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106262270 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14173535259136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14151571118290 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591<¬ð>õž*N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230391937 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2040562237716690 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2051061755417088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2036888561504256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106262270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52320603 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1103 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14173535259136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106262270 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14173535259136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14151571118290 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23473786 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2026410666598400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462958288920576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230391937 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124119674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230381944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124129665 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2036888578543616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14173193912832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14330337533952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106262272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2051062113802752 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2036888578543616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2036888561504256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14151911668292 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14173193912832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚13399Ïc=“4Ÿ*/ _wandbruntime‚11406  _runtime‚11406mÂ…9 *ˆ /loss/FRt1/detection_loss‚0.4791577930263545 1loss/SRqqt2/detection_loss‚0.6020773459351538 %loss/total_loss‚7.123715126495872 _runtime‚11406.287362674 0loss/FRt2/assignment_loss‚0.7923884033954063 /loss/FRt2/detection_loss‚1.1102775185164337 1loss/SRqqt1/detection_loss‚0.8058783370387429 0loss/FBt1/detection_loss‚0.43331946028521434 epoch‚33 trainer/global_step‚13399 % _timestamp‚1.7842842141620057e+09 /loss/FRt1/assignment_loss‚1.561790598981213 3loss/SRqqt2/assignment_loss‚0.43595915041965816 3loss/FBt1/assignment_loss‚0.0036969299917386372 .loss/FBt2/detection_loss‚0.550760043450282 _step‚1104 2loss/SRqqt1/assignment_loss‚0.3443222441876915 2loss/FBt2/assignment_loss‚0.004087301267983845j:ç¡* /loss/FRt1/assignment_loss‚1.561790598981213 3loss/SRqqt2/assignment_loss‚0.43595915041965816 3loss/FBt1/assignment_loss‚0.0036969299917386372 .loss/FBt2/detection_loss‚0.550760043450282 _step‚1104 2loss/SRqqt1/assignment_loss‚0.3443222441876915 2loss/FBt2/assignment_loss‚0.004087301267983845 /loss/FRt1/detection_loss‚0.4791577930263545 1loss/SRqqt2/detection_loss‚0.6020773459351538 %loss/total_loss‚7.123715126495872 _runtime‚11406.287362674 0loss/FRt2/assignment_loss‚0.7923884033954063 /loss/FRt2/detection_loss‚1.1102775185164337 1loss/SRqqt1/detection_loss‚0.8058783370387429 0loss/FBt1/detection_loss‚0.43331946028521434 epoch‚33 trainer/global_step‚13399 % _timestamp‚1.7842842141620057e+09Ð# (¿“¢*‚(:ˆ ¾ˆèÒÉÅø¢ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ€Š²£*‚(:õ ¾ˆèÒ󾥤# disk.nvme0n1.out‚53123.86328125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 3.052541e+06 disk.rbd3.out‚0 disk.dm-0.in‚144850.09765625 disk.sda1.in‚0% disk.nvme0c0n1.out‚53123.86328125 disk.sda.in‚0$ memory_percent‚97.34926223754883& proc.memory.availableMB‚ 434.296875 disk.dm-1.in‚0# disk.nvme0n1.in‚144850.09765625 proc.cpu.threads‚17 network.sent‚ 2.129146e+07* disk./.usagePercent‚36.829090527295236 disk.sda2.in‚0 disk.dm-1.out‚ 453.0546875 disk.rbd1.in‚0.0625! disk.rbd1.out‚112.76220703125 disk.rbd0.in‚0) proc.memory.percent‚17.21494197845459$ disk./.usageGB‚548.7800636291504 disk.sda3.out‚ 453.0546875 disk.sda2.out‚0 disk.sda.out‚ 453.0546875 disk.sda3.in‚0 disk.dm-0.out‚53123.86328125 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144850.09765625# proc.memory.rssMB‚ 2820.49609375 cpu‚26.034023251341477 disk.rbd3.in‚0­‰ÈÆ“¤*‚(:ˆ ͈èÒµô³¢ gpu.0.gpu‚55.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„OT¥*‚(:ü ͈èÒ¸îÓ¤' proc.memory.availableMB‚ 434.3515625# proc.memory.rssMB‚ 2820.49609375 cpu‚26.03362851586262 proc.cpu.threads‚17 disk.dm-1.out‚ 453.6484375 disk.rbd1.in‚0.0625 disk.dm-0.in‚144850.09765625% disk.nvme0c0n1.in‚144850.09765625 disk.sda.out‚ 453.6484375 disk.sda2.out‚0 network.recv‚ 3.057245e+06 disk.rbd3.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0* disk./.usagePercent‚36.829099231551396 disk.rbd3.out‚0! disk.rbd1.out‚112.82470703125$ memory_percent‚97.34892845153809 disk.sda1.in‚0' disk.nvme0c0n1.out‚53128.5244140625 disk.sda3.out‚ 453.6484375 disk.drbd2165.in‚0 network.sent‚ 2.1354464e+07# disk.nvme0n1.in‚144850.09765625% disk.nvme0n1.out‚53128.5244140625 disk.sda1.out‚0$ disk./.usageGB‚548.7801933288574 disk.sda.in‚0 disk.sda2.in‚0" disk.dm-0.out‚53128.5244140625 disk.rbd0.out‚0) proc.memory.percent‚17.21494197845459ÌzOB“¦*‚(:ˆ ܈èÒª‹Ò¢ gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚197.728' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.5456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«µ¾’§*‚(:ý ܈èÒ‚£è¤) disk./.usagePercent‚36.82909257535551 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 434.34765625 cpu‚26.033235721069122 disk.dm-1.in‚0! disk.rbd1.out‚112.82470703125 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 network.recv‚ 3.06059e+06 disk.rbd1.in‚0.0625# disk.nvme0n1.in‚144850.92578125& disk.nvme0n1.out‚53138.98193359375) proc.memory.percent‚17.21494197845459 disk.sda3.out‚ 454.0390625 network.sent‚ 2.1359515e+07 disk.sda1.out‚0( disk.nvme0c0n1.out‚53138.98193359375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚0# disk.dm-0.out‚53138.98193359375 disk.rbd3.in‚0 disk.dm-0.in‚144850.92578125 disk.sda1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2820.49609375" memory_percent‚97.348952293396$ disk./.usageGB‚548.7800941467285 disk.dm-1.out‚ 454.0390625 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144850.92578125 disk.sda.out‚ 454.0390625 jål4¨*/ _wandbruntime‚11445  _runtime‚11445€…ž±Ì©*Æ ' lr-AdamW/pg1‚0.00011874709587535394 ' lr-AdamW/pg2‚0.00011874709587535394 trainer/global_step‚13449 % _timestamp‚1.7842842534369287e+09 _runtime‚11445.562543819 _step‚1105Iop<Ѫ*Ë ' lr-AdamW/pg1‚0.00011874709587535394 ' lr-AdamW/pg2‚0.00011874709587535394 trainer/global_step‚13449 % _timestamp‚1.7842842534369287e+09 _runtime‚11445.562543819 _step‚1105Ñýx×é4«*/ _wandbruntime‚11445  _runtime‚11445‡wŠÙ(¬*P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7842842534379966e+09 _runtime‚11445.563776797 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2058326248743936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14223218126848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106636238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2047790866071604 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23559288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14380552712192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28948019 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚464646530051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52507307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23559291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14222876787712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚13449 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2044103371956224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1106 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450265880373248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14201517101366 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2033589706559488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28948289 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2033589689520128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14380649677824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14201176551476 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106636240 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124565074 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231211291 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231201312 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124575051 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124575051 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2044103388995584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124565074 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2044103388995584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124565074 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2058326248743936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14223218126848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106636238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2047790866071604 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23559288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14380552712192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28948019 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚464646530051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52507307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23559291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14222876787712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚13449 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2044103371956224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1106Òùšç_4®*/ _wandbruntime‚11446  _runtime‚11446O6¾ù¯*‹N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2058474860786176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2044250612985856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124574126 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2044250612985856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231227989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2047938745386560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2047938387797198 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231227989 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106643872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842842540503275e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52511354 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450299309131776 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14224230760960 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2044250630025216 _runtime‚11446.176087148 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52511091 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106643870 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106643872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124584117 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28950318 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14223889414656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106643870 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23561033 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2033736217231360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231217996 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2058474502400512 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2033736200192000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14202528155200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14223889414656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3ñ­úŸ'°*N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124574126 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2044250612985856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231227989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2047938745386560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2047938387797198 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231227989 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106643872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842842540503275e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52511354 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450299309131776 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14224230760960 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2044250630025216 _runtime‚11446.176087148 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52511091 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106643870 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106643872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124584117 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28950318 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14223889414656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106643870 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23561033 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2033736217231360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231217996 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2058474502400512 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2033736200192000 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14202528155200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14223889414656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2058474860786176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2044250612985856ÓåÈz¶4±*/ _wandbruntime‚11446  _runtime‚11446ŠO3²*‰ 0loss/FRt2/assignment_loss‚0.8176625230490026 2loss/SRqqt1/assignment_loss‚0.3351108653726036 4loss/FBt2/assignment_loss‚0.00012973785648573433 %loss/total_loss‚7.151834460066716 _step‚1108 3loss/SRqqt2/assignment_loss‚0.46432403839092234 1loss/FBt1/assignment_loss‚0.01979909483384816 .loss/FRt2/detection_loss‚1.130910795467076 /loss/FBt1/detection_loss‚0.3860947607960649 0loss/FBt2/detection_loss‚0.38255630174535327 trainer/global_step‚13449 _runtime‚11446.178522254 /loss/FRt1/detection_loss‚0.5144352982825388 1loss/SRqqt1/detection_loss‚1.0015349817775383 0loss/FRt1/assignment_loss‚1.4638853830610004 1loss/SRqqt2/detection_loss‚0.6353906794342816 epoch‚33 % _timestamp‚1.7842842540531893e+09K«ó³*Ž % _timestamp‚1.7842842540531893e+09 0loss/FRt2/assignment_loss‚0.8176625230490026 2loss/SRqqt1/assignment_loss‚0.3351108653726036 4loss/FBt2/assignment_loss‚0.00012973785648573433 %loss/total_loss‚7.151834460066716 _step‚1108 3loss/SRqqt2/assignment_loss‚0.46432403839092234 1loss/FBt1/assignment_loss‚0.01979909483384816 .loss/FRt2/detection_loss‚1.130910795467076 /loss/FBt1/detection_loss‚0.3860947607960649 0loss/FBt2/detection_loss‚0.38255630174535327 trainer/global_step‚13449 _runtime‚11446.178522254 /loss/FRt1/detection_loss‚0.5144352982825388 1loss/SRqqt1/detection_loss‚1.0015349817775383 0loss/FRt1/assignment_loss‚1.4638853830610004 1loss/SRqqt2/detection_loss‚0.6353906794342816 epoch‚33Ô‚Ñ0´*‚(:„ ëˆèÒÅØÌ£ gpu.0.gpu‚9.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/bĵ*‚(:÷ ëˆèÒ÷ùí¤ network.recv‚ 3.064273e+06) disk./.usagePercent‚36.82927024458427 disk.rbd0.in‚0 disk.sda.out‚ 454.76953125$ disk.nvme0n1.out‚53146.275390625 disk.sda2.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2820.49609375$ disk./.usageGB‚548.7827415466309 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 144851.640625) proc.memory.percent‚17.21494197845459! disk.dm-0.out‚53146.275390625 disk.sda1.out‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚53146.275390625 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 454.76953125 disk.sda1.in‚0( proc.memory.availableMB‚ 434.10546875 cpu‚26.032822815305142 proc.cpu.threads‚17 network.sent‚ 2.1364572e+07# disk.nvme0c0n1.in‚ 144851.640625 disk.sda3.out‚ 454.76953125 disk.drbd2165.out‚0$ memory_percent‚97.35043048858643! disk.nvme0n1.in‚ 144851.640625 disk.dm-1.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚112.82470703125ûT’»“¶*‚(:ˆ úˆèҌ֩¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w•T¹ý·*‚(:ò úˆèÒáÚ¤ cpu‚26.03236847944641" disk./.usageGB‚548.78271484375 disk.sda.in‚0 disk.dm-1.out‚ 454.9921875) disk./.usagePercent‚36.82926845253153 disk.sda3.in‚0 disk.nvme0n1.in‚ 144852.46875& disk.nvme0n1.out‚53156.17333984375( disk.nvme0c0n1.out‚53156.17333984375& proc.memory.availableMB‚ 434.171875 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 144852.46875 disk.sda.out‚ 454.9921875 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 144852.46875 disk.drbd2165.out‚0 network.recv‚ 3.068638e+06 disk.sda3.out‚ 454.9921875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2820.49609375 network.sent‚ 2.1427582e+07 disk.rbd3.out‚0# disk.dm-0.out‚53156.17333984375$ memory_percent‚97.35002517700195) proc.memory.percent‚17.21494197845459 disk.sda2.in‚0 disk.sda2.out‚0! disk.rbd1.out‚112.94970703125þ‘F04¸*/ _wandbruntime‚11485  _runtime‚11485ñµÅ¦Ì¹*Æ ' lr-AdamW/pg1‚0.00011710792378558639 ' lr-AdamW/pg2‚0.00011710792378558639 trainer/global_step‚13499 % _timestamp‚1.7842842931223865e+09 _runtime‚11485.248002748 _step‚1109!åѺ*Ë _runtime‚11485.248002748 _step‚1109 ' lr-AdamW/pg1‚0.00011710792378558639 ' lr-AdamW/pg2‚0.00011710792378558639 trainer/global_step‚13499 % _timestamp‚1.7842842931223865e+09Õ ƒtµ4»*/ _wandbruntime‚11485  _runtime‚11485VÛqŽó&¼*P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451937318299648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14431940176384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23646538 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451937299687424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2065739354105856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2051465423437824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232047343 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2065738995727360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14273913628672 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2051465423437824 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107017838 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107017838 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14431855793664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125029503 XKDeviceStatsMonitor.on_train_batch_start/inactive_sçÃéAplit.large_pool.allocated‚23646541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2065739354105856 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 çƒZ (½*†P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125029503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29052449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52698987 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14252133588274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23646538 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451937299687424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2065739354105856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2051465423437824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232047343 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2065738995727360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14273913628672 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2051465423437824 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107017838 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107017838 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14431855793664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125029503 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23646541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2065739354105856 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451937318299648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14431940176384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125038569 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2051612681506816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125038569 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14253144642108 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232064041 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2065887607769600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107025472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚451970728445952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 232064041 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 trainer/global_step‚13499 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232054048 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52702753 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107025470 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125028578 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2055314895467068 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842842937351775e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107025472 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107025470 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52703028 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125028578 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29054470 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14432876185600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466403709716480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23648283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23648286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2055314537877706 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14432962658304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _step‚1111T)!'À*N FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2065887607769600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107025472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚451970728445952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 232064041 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 trainer/global_step‚13499 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232054048 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52702753 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107025470 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125028578 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2055314895467068 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842842937351775e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107025472 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107025470 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52703028 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125028578 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29054470 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14432876185600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466403709716480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23648283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23648286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2055314537877706 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14432962658304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _step‚1111 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14274926262784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125038569 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2051612681506816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125038569 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14253144642108 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232064041×Qäöt4Á*/ _wandbruntime‚11485  _runtime‚11485)…®“ Â*… /loss/FBt2/detection_loss‚0.7016809279907887 %loss/total_loss‚7.380681706084765 % _timestamp‚1.7842842937382822e+09 /loss/FRt1/assignment_loss‚1.548322280532463 2loss/FBt2/assignment_loss‚0.041393656865144644 .loss/FRt2/detection_loss‚1.237152566774379 epoch‚33 2loss/SRqqt1/assignment_loss‚0.3245856090721497 2loss/FBt1/assignment_loss‚0.013458160841050747 /loss/FRt1/detection_loss‚0.4977237120439919 trainer/global_step‚13499 _runtime‚11485.863621048 _step‚1112 2loss/SRqqt2/assignment_loss‚0.4406155589205945 1loss/SRqqt2/detection_loss‚0.5448363710056046 /loss/FBt1/detection_loss‚0.4452272255662234 0loss/FRt2/assignment_loss‚0.7732308010068115 1loss/SRqqt1/detection_loss‚0.8124548354655627Ùä]Ã*Š _step‚1112 2loss/SRqqt2/assignment_loss‚0.4406155589205945 1loss/SRqqt2/detection_loss‚0.5448363710056046 /loss/FBt1/detection_loss‚0.4452272255662234 0loss/FRt2/assignment_loss‚0.7732308010068115 1loss/SRqqt1/detection_loss‚0.8124548354655627 /loss/FBt2/detection_loss‚0.7016809279907887 %loss/total_loss‚7.380681706084765 % _timestamp‚1.7842842937382822e+09 /loss/FRt1/assignment_loss‚1.548322280532463 2loss/FBt2/assignment_loss‚0.041393656865144644 .loss/FRt2/detection_loss‚1.237152566774379 epoch‚33 2loss/SRqqt1/assignment_loss‚0.3245856090721497 2loss/FBt1/assignment_loss‚0.013458160841050747 /loss/FRt1/detection_loss‚0.4977237120439919 trainer/global_step‚13499 _runtime‚11485.863621048ØÎFõ“Ä*‚(:ˆ ‰‰èÒ†úŸ¢ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Y9÷]Å*‚(:ú ‰‰èÒТ‚¥& disk.nvme0n1.out‚53163.31298828125 disk.rbd0.out‚0 disœê¹~¨k.dm-1.out‚ 455.328125 proc.cpu.threads‚17) disk./.usagePercent‚36.82926845253153# disk.nvme0n1.in‚144853.18359375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-0.in‚144853.18359375 disk.sda2.out‚0' proc.memory.availableMB‚ 434.1796875) proc.memory.percent‚17.21494197845459" disk./.usageGB‚548.78271484375 disk.sda.out‚ 455.328125 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144853.18359375 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 network.sent‚ 2.1432733e+07 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2820.49609375 network.recv‚ 3.072393e+06 disk.rbd3.in‚0$ memory_percent‚97.34997749328613 disk.sda1.in‚0( disk.nvme0c0n1.out‚53163.31298828125 disk.sda3.in‚0 disk.sda3.out‚ 455.328125! disk.rbd1.out‚112.94970703125 cpu‚26.031978211086148# disk.dm-0.out‚53163.312988281258ëÍ£“Æ*‚(:ˆ ˜‰èÒŽ³¢ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚273.148' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öñ=Ç*‚(:ö ˜‰èÒÞ•ï¤ network.recv‚ 3.076649e+06 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0# disk.nvme0n1.in‚144853.18359375 disk.sda.out‚ 455.71875 disk.sda2.in‚0! disk.rbd1.out‚113.01220703125( disk./.usagePercent‚36.8292771567877# disk.dm-0.out‚53168.48193359375 disk.drbd2165.in‚0) proc.memory.percent‚17.21494197845459 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 cpu‚26.03159102700897 proc.cpu.threads‚17 network.sent‚ 2.1495925e+07# disk./.usageGB‚548.782844543457 disk.dm-0.in‚144853.18359375& disk.nvme0n1.out‚53168.48193359375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.in‚144853.18359375( disk.nvme0c0n1.out‚53168.48193359375 disk.sda3.out‚ 455.71875 disk.dm-1.out‚ 455.71875$ memory_percent‚97.34959602355957# proc.memory.rssMB‚ 2820.49609375' proc.memory.availableMB‚ 434.2421875¶u`ÚžÈ*‚(:“ §‰èÒ“Æã gpu.0.gpu‚51.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚208.464' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.692800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jù²{É*‚(:ø §‰èÒýòФ# disk.nvme0n1.out‚53175.75390625 disk.sda3.out‚ 456.01171875 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚26.031162106344656% disk.nvme0c0n1.in‚144854.01171875 disk.sda.out‚ 456.01171875 network.recv‚ 3.080218e+06 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.dm-0.in‚144854.01171875 disk.sda1.out‚0% disk.nvme0c0n1.out‚53175.75390625 disk.sda.in‚0$ memory_percent‚97.34964370727539$ disk./.usageGB‚548.7828254699707 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-0.out‚53175.75390625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2820.49609375 disk.sda3.in‚0) proc.memory.percent‚17.21494197845459 network.sent‚ 2.1500978e+07# disk.nvme0n1.in‚144854.01171875 disk.drbd2165.in‚0& proc.memory.availableMB‚ 434.234375) disk./.usagePercent‚36.82927587675003 disk.dm-1.out‚ 456.01171875! disk.rbd1.out‚113.01220703125‘òËá4Ê*/ _wandbruntime‚11525  _runtime‚11525-’ÌË*Æ ' lr-AdamW/pg1‚0.00011547640937077192 ' lr-AdamW/pg2‚0.00011547640937077192 trainer/global_step‚13549 % _timestamp‚1.7842843331148686e+09 _runtime‚11525.240479077 _step‚1113™4”ÑÌ*Ë ' lr-AdamW/pg2‚0.00011547640937077192 trainer/global_step‚13549 % _timestamp‚1.7842843331148686e+09 _runtime‚11525.240479077 _step‚1113 ' lr-AdamW/pg1‚0.00011547640937077192Ù«"àú4Í*/ _wandbruntime‚11525  _runtime‚11525_\¡ý(Î*P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2062543523821870 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14483251068928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚11525.241717345 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2058827474919424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚453608737613824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29156178 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107399440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2073151742710784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125473978 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125483955 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2048240773746688 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2073152101089280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107399438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2058827491958784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468091887522816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14302750075182 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14302409525292 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚13549 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2062543166232620 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23733788 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52889966 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2073151742710784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14324267791360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468092007294976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125483955 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453608756226048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14483149908992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2073151742710784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14324267791360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468092007294976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125483955 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453608756226048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14483149908992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14483251068928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚11525.241717345 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2058827474919424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚453608737613824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29156178 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107399440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2073151742710784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125473978 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125483955 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2048240773746688 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2073152101089280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107399438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2058827491958784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468091887522816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14302750075182 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14302409525292 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚13549 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2062543166232620 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23733788 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52889966 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977ÚБq\4Ð*/ _wandbruntime‚11525  _runtime‚11525Ÿ5¢¢|Ñ*‹N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468126332216832 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚453642166372352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14303761129016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125493021 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2073300354753024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2062690687958214 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14303420579014 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29158418 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52893681 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚13549 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2058974715949056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23735536 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2062691045547576 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125483030 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2073299996367360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14484165844480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14484262802944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14325621764608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468126447787520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107407072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107407070 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 %Ë$W• _timestamp‚1.7842843337304044e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1115 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232900093 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23735533 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14303761129016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125493021 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2073300354753024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2062690687958214 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14303420579014 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29158418 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52893681 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚13549 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2058974715949056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23735536 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2062691045547576 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125483030 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2073299996367360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14484165844480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14484262802944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14325621764608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468126447787520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107407072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107407070 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7842843337304044e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1115 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232900093 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23735533 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2066189543440384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53080664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2080564848072704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233719447 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2080564848072704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107781038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125928430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2080564489694208 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2066189543440384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455280175540224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983&È´á*…P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233719447 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2080564848072704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107781038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125928430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2080564489694208 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2066189543440384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455280175540224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125928430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2066189543440384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53080664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2080564848072704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496Þç÷»÷4â*/ _wandbruntime‚11566  _runtime‚11566rž'ã*ŠN YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107788670 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125947473 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2070066838038722 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233736145 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107788670 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2066336767430656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14375975920128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53084221 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14354377615924 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233726152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125947473 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455313604298752 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2080713101736448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53084221 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14354377615924 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233726152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125947473 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455313604298752 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2080713101736448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2070066838038722 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233736145 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107788670 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2066336767430656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14375975920128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ßÝ'ì4å*/ _wandbruntime‚11566  _runtime‚11566÷M)æ*ˆ 0loss/FRt1/assignment_loss‚1.5233379487807934 2loss/SRqqt1/assignment_loss‚0.3591555760384857 2loss/SRqqt2/assignment_loss‚0.4324516831349351 /loss/FRt1/detection_loss‚0.4961748936680988 /loss/FRt2/detection_loss‚1.2272223394686217 epoch‚33 _step‚1120 2loss/FBt1/assignment_loss‚0.006000640787294889 1loss/SRqqt2/detection_loss‚0.5421141456129349 % _timestamp‚1.7842843743199675e+09 0loss/FRt2/assignment_loss‚0.7192252589507662 1loss/FBt2/assignment_loss‚0.25935516768874317 1loss/SRqqt1/detection_loss‚0.9712158868159709 /loss/FBt1/detection_loss‚0.4377525800620286 0loss/FBt2/detection_loss‚0.49165688922221307 &loss/total_loss‚7.4656630102308865 trainer/global_step‚13599 _runtime‚11566.445323441U‘ÑCç* 2loss/FBt1/assignment_loss‚0.006000640787294889 1loss/SRqqt2/detection_loss‚0.5421141456129349 % _timestamp‚1.7842843743199675e+09 0loss/FRt2/assignment_loss‚0.7192252589507662 1loss/FBt2/assignment_loss‚0.25935516768874317 1loss/SRqqt1/detection_loss‚0.9712158868159709 /loss/FBt1/detection_loss‚0.4377525800620286 0loss/FBt2/detection_loss‚0.49165688922221307 &loss/total_loss‚7.4656630102308865 trainer/global_step‚13599 _runtime‚11566.445323441 0loss/FRt1/assignment_loss‚1.5233379487807934 2loss/SRqqt1/assignment_loss‚0.3591555760384857 2loss/SRqqt2/assignment_loss‚0.4324516831349351 /loss/FRt1/detection_loss‚0.4961748936680988 /loss/FRt2/detection_loss‚1.2272223394686217 epoch‚33 _step‚1120à¶MŸ4“è*‚(:ˆ ã‰èÒ±Œ£¥ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D_Ñé*‚(:û ã‰èÒï—¨ disk.dm-0.in‚144856.26953125( disk.nvme0c0n1.out‚53242.37744140625# memory_percent‚97.3606824874878 disk.dm-1.in‚0) proc.memory.percent‚17.22583770751953) disk./.usagePercent‚36.82899478047743# disk./.usageGB‚548.778636932373 disk.dm-1.out‚ 465.6953125 disk.sda1.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 432.42578125 network.recv‚ 3.094911e+06# disk.nvme0n1.in‚144856.26953125& disk.nvme0n1.out‚53242.37744140625 disk.sda.out‚ 465.6953125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 465.6953125! disk.rbd1.out‚113.13720703125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 cpu‚26.034059744839464 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚53242.37744140625 disk.rbd0.out‚0 disk.sda.in‚0 proc.memory.rssMB‚ 2822.28125 network.sent‚ 2.1592201e+07 disk.rbd1.in‚0.0625% disk.nvme0c0n1.in‚144856.26953125 disk.sda3.in‚0žV¸4ê*/ _wandbruntime‚11583  _runtime‚11583P~rá ë*ðz :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2528504729270935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6826537847518921 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25915011763572693 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6682184934616089 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963869452476501 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9480319023132324 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 &particle/f_score‚0.630623459815979 particle/accuracy_2_of_5‚1 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.634613573551178 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7803069353103638 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.512499988079071 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2987031042575836 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7653817534446716 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8982940912246704 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚13599 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6968355774879456 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7839381098747253 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04188719391822815 *jet/accuracy_1_of_3‚0.9339394569396973 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2407439947128296 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3080880641937256 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20176129043102264 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.01866452395915985 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8630976676940918 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03915509581565857 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2694207429885864 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9894846677780151 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4923876225948334 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5558758974075317 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6928528547286987 epoch‚33 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 7 Purity/2FRt*SRqqtÅŽ“Õ•10FBt/FRt_purity‚0.2399977445602417 +jet/accuracy_2_of_2‚0.16390913724899292 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22505076229572296 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47124093770980835 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6345133781433105 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2695167660713196 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.963261785970785 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6708521842956543 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6133809685707092 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7233561277389526 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04894852638244629 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32263892889022827 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3083396553993225 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7761486768722534 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518516421318054 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7319875955581665 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6521479487419128 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1880442351102829 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.018790453672409058 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27989232540130615 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966000914573669 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4283585846424103 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 jet/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 jet/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2334987074136734 particle/accuracy_2_of_4‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.71875 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_4‚0.07950375974178314 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09354793280363083 +particle/sensitivity‚0.6204062104225159 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.298227459192276 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2614624500274658 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7196482419967651 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7977105379104614 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_4‚0.8182652592658997 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6492064595222473 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5351994037628174 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6920377016067505 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6916992664337158 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2902423143386841 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28933408856391907 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7810013294219971 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416666328907013 6validation_average_jet_accuracy‚0.3908827602863312 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6357547044754028 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.8023970723152161 /particle/accuracy_3_of_4‚0.9979296326637268 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_3_of_3‚0.06516531854867935 *particle/specificity‚0.905062198638916 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.634613573551178 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.715103268623352 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021078182384371758 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_2_of_3‚0.5817888975143433 6"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435586810112 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19962474703788757 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _runtime‚11583.501313565 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8055555820465088 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8741238713264465 *jet/accuracy_1_of_4‚0.9775999784469604 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416865348816 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ,jet/accuracy_4_of_4‚0.013899499550461769 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.75 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.789984941482544 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6590997576713562 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7647204399108887 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.01693308912217617 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7920348644256592 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04870768263936043 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2987031042575836 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 )jet/accuracy_1_of_1‚0.351154088973999 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35594940185546875 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3083275258541107 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27154049277305603 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8741238713264465 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30558207631111145 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6255820393562317 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7397734522819519 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941205382347107 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6242161989212036 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.628933846950531 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5511541962623596 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6248992681503296 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5678322315216064 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3781155049800873 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2453640252351761 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6851754784584045 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48665231466293335 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7839381098747253 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6900926232337952 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7134732007980347 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021078182384371758 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2664405405521393 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8694825172424316 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21306507289409637 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7166830897331238 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8055555820465088 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08034869283437729 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3516806364059448 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7695181965827942 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.843936026096344 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 _step‚1121 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5532206296920776 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.01708638295531273 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3760962188243866 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6334255933761597 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842843913750477e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9310819506645203 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6586048603057861 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_3_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344íUfÖ{=ì*õz :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47124093770980835 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6345133781433105 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2695167660713196 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.963261785970785 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6708521842956543 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6133809685707092 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7233561277389526 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04894852638244629 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32263892889022827 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3083396553993225 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7761486768722534 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518516421318054 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7319875955581665 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6521479487419128 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1880442351102829 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.018790453672409058 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27989232540130615 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966000914573669 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4283585846424103 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 jet/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 jet/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2334987074136734 particle/accuracy_2_of_4‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.71875 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_4‚0.07950375974178314 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09354793280363083 +particle/sensitivity‚0.6204062104225159 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.298227459192276 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2614624500274658 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7196482419967651 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7977105379104614 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_4‚0.8182652592658997 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6492064595222473 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5351994037628174 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6920377016067505 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6916992664337158 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2902423143386841 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28933408856391907 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7810013294219971 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416666328907013 6validation_average_jet_accuracy‚0.3908827602863312 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6357547044754028 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.8023970723152161 /particle/accuracy_3_of_4‚0.9979296326637268 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_3_of_3‚0.06516531854867935 *particle/specificity‚0.905062198638916 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.634613573551178 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.715103268623352 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021078182384371758 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_2_of_3‚0.5817888975143433 6"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435586810112 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19962474703788757 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _runtime‚11583.501313565 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8055555820465088 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8741238713264465 *jet/accuracy_1_of_4‚0.9775999784469604 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416865348816 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ,jet/accuracy_4_of_4‚0.013899499550461769 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.75 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.789984941482544 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6590997576713562 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7647204399108887 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.01693308912217617 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7920348644256592 jet/accuracy_4_of_5‚0 particle/accuracy_4_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04870768263936043 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2987031042575836 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 )jet/accuracy_1_of_1‚0.351154088973999 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35594940185546875 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3083275258541107 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27154049277305603 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8741238713264465 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30558207631111145 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6255820393562317 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7397734522819519 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991950988769531 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941205382347107 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6242161989212036 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.628933846950531 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5511541962623596 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6248992681503296 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5678322315216064 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3781155049800873 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2453640252351761 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6851754784584045 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48665231466293335 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7839381098747253 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6900926232337952 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7134732007980347 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021078182384371758 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2664405405521393 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8694825172424316 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21306507289409637 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7166830897331238 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8055555820465088 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08034869283437729 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3516806364059448 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7695181965827942 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.843936026096344 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 _step‚1121 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5532206296920776 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.01708638295531273 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3760962188243866 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6334255933761597 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842843913750477e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9310819506645203 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6586048603057861 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_3_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2528504729270935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6826537847518921 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25915011763572693 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6682184934616089 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963869452476501 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9480319023132324 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 &particle/f_score‚0.630623459815979 particle/accuracy_2_of_5‚1 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.634613573551178 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7803069353103638 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.512499988079071 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2987031042575836 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7653817534446716 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8982940912246704 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚13599 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6968355774879456 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7839381098747253 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04188719391822815 *jet/accuracy_1_of_3‚0.9339394569396973 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2407439947128296 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3080880641937256 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20176129043102264 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.01866452395915985 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8630976676940918 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03915509581565857 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2694207429885864 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9894846677780151 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4923876225948334 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5558758974075317 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6928528547286987 epoch‚33 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2399977445602417 +jet/accuracy_2_of_2‚0.16390913724899292 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22505076229572296áþϵ;“í*‚(:ˆ ò‰èÒÎý„¥ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m9È î*‚(:þ ò‰èÒ­á˦# disk.nvme0n1.in‚144857.09765625 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 466.35546875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 466.35546875 disk.dm-1.out‚ 466.35546875 proc.memory.rssMB‚ 2822.28125$ memory_percent‚97.35941886901855' proc.memory.availableMB‚ 432.6328125$ disk./.usageGB‚548.7826538085938 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚144857.09765625) proc.memory.percent‚17.22583770751953 disk.sda1.out‚0% disk.nvme0c0n1.in‚144857.09765625 cpu‚26.03496829285709 network.sent‚ 2.1642276e+07 network.recv‚ 3.099187e+06& disk.nvme0n1.out‚53252.89111328125( disk.nvme0c0n1.out‚53252.89111328125 disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.82926435641099 disk.rbd0.out‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚113.19970703125 disk.rbd3.out‚0# disk.dm-0.out‚53252.89111328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0e´ œ’ï*‚(:‡ ŠèÒ°ëÝ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+œ¬Oøð*‚(:í ŠèÒÐí¤ proc.memory.rssMB‚ 2822.28125) proc.memory.percent‚17.22583770751953 cpu‚26.034566746108048 disk.drbd2165.out‚0 disk.sda3.in‚0# memory_percent‚97.3597764968872! disk.nvme0c0n1.in‚ 144857.8125 disk.rbd3.out‚0$ disk./.usageGB‚548.7871017456055 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 2.1699563e+07) disk./.usagePercent‚36.82956286119591( disk.nvme0c0n1.out‚53269.38623046875 disk.sda3.out‚ 468.359375 disk.dm-0.in‚ 144857.8125& disk.nvme0n1.out‚53269.38623046875# disk.dm-0.out‚53269.38623046875 disk.nvme0n1.in‚ 144857.8125 disk.rbd1.in‚0.0625 disk.rbd3.in‚0( proc.memory.availableMB‚ 432.57421875 network.recv‚ 3.103636e+06 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 468.359375 proc.cpu.threads‚17 disk.sda.out‚ 468.359375! disk.rbd1.out‚113.19970703125*õ‡>4ñ*/ _wandbruntime‚11622  _runtime‚11622éVåÌò*Æ ' lr-AdamW/pg1‚0.00011223677222392275 ' lr-AdamW/pg2‚0.00011223677222392275 trainer/global_step‚13649 % _timestamp‚1.7842844306808727e+09 _runtime‚11622.806484479 _step‚1122+·mšÑó*Ë ' lr-AdamW/pg1‚0.00011223677222392275 ' lr-AdamW/pg2‚0.00011223677222392275 trainer/global_step‚13649 % _timestamp‚1.7842844306808727e+09 _runtime‚11622.806484479 _step‚1122âwÄ j4ô*/ _wandbruntime‚11622  _runtime‚11622í‘|axõ*€P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚13649 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2075481161572352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53306123 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/ove;t/Ž$rsize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234746487 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126463181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234736508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2079221849332945 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14443392000512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2075481144532992 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2089924894912000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14443733339648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2089924536533504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚73896960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23910808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14603714246144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2089924894912000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚471571332321792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14603788143104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2075481161572352 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚1123 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23910811 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471571239812608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2089924536533504 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126453204 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2075481144532992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456967525566464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108283304 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108283306¿T½« (ö*…P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14443392000512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2075481144532992 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2089924894912000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14443733339648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2089924536533504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚73896960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23910808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14603714246144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2089924894912000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚471571332321792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14603788143104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2075481161572352 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚1123 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23910811 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471571239812608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2089924536533504 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126453204 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2075481144532992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456967525566464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108283304 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108283306 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚13649 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2075481161572352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53306123 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234746487 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126463181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234736508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2079221849332945 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976ã‹eO4÷*/ _wandbruntime‚11623  _runtime‚11623òL¿'ø*‹N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2079369371058539 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2075628385562624 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2075628385562624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53309666 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234763185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2090072790190080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2090073148575744 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126472247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108290938 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234763185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23912556 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14604735031296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14444404627456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29397378 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2064947014819840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126472247 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14422373278059 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1124 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108290938 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14444404627456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126462256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29397113 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚471605689356288 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2079369728647901 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23912553 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚471605792344064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14422713828061 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2075628402601984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14444745973760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53309934 trainer/global_step‚13649 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14444745973760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14604819406848 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2090073148575744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457000954324992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234753192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ÃY€ÿ ù*N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23912553 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚471605792344064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14422713828061 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2075628402601984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14444745973760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53309934 trainer/global_step‚13649 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14444745973760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14604819406848 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2090073148575744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457000954324992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234753192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2079369371058539 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_b¶1Õ~atch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2075628385562624 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2075628385562624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53309666 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234763185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2090072790190080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2090073148575744 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126472247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108290938 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234763185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23912556 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14604735031296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14444404627456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29397378 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2064947014819840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126472247 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14422373278059 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1124 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108290938 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14444404627456 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126462256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29397113 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚471605689356288 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2079369728647901ä ×*´4ú*/ _wandbruntime‚11623  _runtime‚11623ÛÏZû*‰ 2loss/SRqqt2/assignment_loss‚0.4627206344156801 3loss/FBt2/assignment_loss‚0.0006545435208296256 /loss/FRt2/detection_loss‚1.2068900446482977 1loss/SRqqt2/detection_loss‚0.6065929426291226 % _timestamp‚1.7842844312951014e+09 _runtime‚11623.420456255 0loss/FRt1/assignment_loss‚1.5045990710814274 2loss/FBt1/assignment_loss‚0.010619602379869586 1loss/SRqqt1/detection_loss‚0.7883424541983834 /loss/FBt1/detection_loss‚0.3961542832664508 %loss/total_loss‚7.301152771802236 _step‚1125 /loss/FRt1/detection_loss‚0.5502768685565037 epoch‚34 3loss/SRqqt1/assignment_loss‚0.32487632341892353 /loss/FBt2/detection_loss‚0.6713205985207524 trainer/global_step‚13649 0loss/FRt2/assignment_loss‚0.7781054051659948Ñâãmü*Ž 1loss/SRqqt1/detection_loss‚0.7883424541983834 /loss/FBt1/detection_loss‚0.3961542832664508 %loss/total_loss‚7.301152771802236 _step‚1125 /loss/FRt1/detection_loss‚0.5502768685565037 epoch‚34 3loss/SRqqt1/assignment_loss‚0.32487632341892353 /loss/FBt2/detection_loss‚0.6713205985207524 trainer/global_step‚13649 0loss/FRt2/assignment_loss‚0.7781054051659948 2loss/SRqqt2/assignment_loss‚0.4627206344156801 3loss/FBt2/assignment_loss‚0.0006545435208296256 /loss/FRt2/detection_loss‚1.2068900446482977 1loss/SRqqt2/detection_loss‚0.6065929426291226 % _timestamp‚1.7842844312951014e+09 _runtime‚11623.420456255 0loss/FRt1/assignment_loss‚1.5045990710814274 2loss/FBt1/assignment_loss‚0.010619602379869586å `´´ý*‚(:’ ŠèÒ•™Á¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚268.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.767999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸fÚVùþ*‚(:î ŠèÒк‰¥* disk./.usagePercent‚36.829704945377415& disk.nvme0n1.out‚53292.98974609375 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 144858.640625( disk.nvme0c0n1.out‚53292.98974609375 disk.sda2.out‚0$ memory_percent‚97.36273288726807 disk.sda.in‚0 proc.memory.rssMB‚ 2822.28125 proc.cpu.threads‚17 disk.dm-0.in‚ 144858.640625 network.sent‚ 2.1704582e+07! disk.nvme0n1.in‚ 144858.640625 disk.drbd2165.in‚0! disk.rbd1.out‚113.32470703125 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚469.6875 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 432.08984375 cpu‚26.03411430065806$ disk./.usageGB‚548.7892189025879 disk.rbd0.out‚0 disk.sda3.out‚469.6875 disk.dm-1.out‚469.6875) proc.memory.percent‚17.22583770751953 network.recv‚ 3.107005e+06# disk.dm-0.out‚53292.98974609375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0[ êX“ÿ*‚(:ˆ ŸŠèÒ­šŸ¤ gpu.0.gpu‚87.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?Ø\þ€+‚(:ó ŸŠèÒЇƥ disk.sda1.out‚0# disk.nvme0c0n1.in‚ 144858.640625 cpu‚26.033726022159737$ disk.nvme0n1.out‚53302.443359375 disk.rbd0.out‚0 disk.sda2.out‚0! disk.rbd1.out‚113.32470703125 disk.dm-0.in‚ 144858.640625 network.sent‚ 2.1767585e+07! disk.nvme0n1.in‚ 144858.640625( proc.memory.availableMB‚ 432.38671875 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0# memory_percent‚97.3609209060669 disk.drbd2165.out‚0! disk.dm-0.out‚53302.443359375) proc.memory.percent‚17.22583770751953 proc.cpu.threads‚17) disk./.usagePercent‚36.82928739708907& disk.nvme0c0n1.out‚53302.443359375 disk.sda3.out‚ 474.12109375 proc.memory.rssMB‚ 2822.28125 network.recv‚ 3.111586e+06 disk.sda2.in‚0 disk.dm-1.out‚ 474.12109375 disk.rbd3.in‚0$ disk./.usageGB‚548.7829971313477 disk.sda1.in‚0 disk.sda.out‚ 474.12109375 disk.sda3.in‚0 disk.dm-1.in‚0bÎcœ+‚(:‘ ®ŠèÒú¡Ø¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.079' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.015800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.“ysñ‚+‚(:æ ®ŠèÒÛ…ý¤) disk./.usagePercent‚36.82884655211511 disk.dm-1.in‚0 disk.drbd2165.out‚0$ memory_percent‚97.36318588256836 disk.nvme0n1.out‚ 53313.6875 network.sent‚ 2.1772639e+07 disk.dm-1.out‚ 474.9765625! disk.rbd1.out‚113.38720703125 disk.dm-0.out‚ 53313.6875# disk.nvme0n1.in‚144859.35546875 proc.cpu.threads‚17$ disk./.usageGB‚548.7764282226562 disk.rbd0.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 432.015625) proc.memory.percent‚17.22583770751953 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚ 474.9765625 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚144859.35546875 proc.memory.rssMB‚ 2822.28125 network.recv‚ 3.114848e+06% disk.nvme0c0n1.in‚144859.35546875! disk.nvme0c0n1.out‚ 53313.6875 disk.sda3.out‚ 474.9765625 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd3.in‚0 cpu‚26.033322176567395Y^»¤4ƒ+/ _wandbruntime‚11662  _runtime‚11662‚Ý…É„+à trainer/global_step‚13699 $ _timestamp‚1.784284470626357e+09 _runtime‚11662.751951383 _step‚1126 & lr-AdamW/pg1‚0.0001106288575677281 & lr-AdamW/pg2‚0.0001106288575677281 3ž Î…+È _runtime‚11662.751951383 _step‚1126 & lr-AdamW/pg1‚0.0001106288575677281 & lr-AdamW/pg2‚0.0001106288575677281 trainer/global_step‚13699 $ _timestamp‚1.784284470626357e+09æoœð4†+/ _wandbruntime‚11662  _runtime‚11662§´> (‡+ÿO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 _step‚1127 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108664906 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126907656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126907656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458638982105088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2082843196014592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23998058 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2086597999413453 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2072126020702208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2082843213053952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53497533 trainer/global_step‚13699 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2082843213053952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14494087502336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2097337641895424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23998061 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14655055119872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235582539 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2097337283516928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29499217 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842844706274042e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126917633 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14472319261135 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2082843196014592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14494087502336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473294018612736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14471978711245 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473294117413376 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2072126037741568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚11662.75315732 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108664906 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108664904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2086597999413453 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2072126020702208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2082843213053952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53497533 trainer/global_step‚13699 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2082843213053952 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14494087502336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2097337641895424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23998061 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14655055119872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235582539 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2097337283516928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29499217 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842844706274042e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126917633 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14472319261135 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2082843196014592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14494087502336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473294018612736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14471978711245 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473294117413376 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2072126037741568 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚11662.75315732 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_po2† ¹ ol.current‚ 341339136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108664906 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108664904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53501079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842844712410865e+09 _runtime‚11663.366928787 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2097485895559168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14472989764967 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458672410863616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235589244 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2082990454083584 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2097485537173504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14495100129280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2072272531374080 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235599237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14495441475584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108672536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126916708 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29501549 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2082990454083584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29501276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235599237 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2086745878728409 _step‚1128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108672536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53501355 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126916708 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108672538 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126926699 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473328462798848 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14656161214464 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2072272548413440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23999803 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126926699 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235589244 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476Ov1'‹+N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473328572078080 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14495441475584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53501079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842844712410865e+09 _runtime‚11663.366928787 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2097485895559168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14472989764967 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458672410863616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235589244 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2082990454083584 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2097485537173504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14495100129280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2072272531374080 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235599237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14495441475584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108672536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126916708 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29501549 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2082990454083584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29501276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235599237 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2086745878728409 _step‚1128 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108672536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53501355 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126916708 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108672538 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126926699 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473328462798848 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14656161214464 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2072272548413440 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23999803 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126926699 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235589244 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458672392251392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14473330314969 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2104750030500352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24085308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29604040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2093974507083211 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2104750030500352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14545124343296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109046504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2079451554295808 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2090205264535552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14706447887360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2079451571335168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127372085 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127362108 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460310420031488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53689087 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236408612 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2090205264535552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 5Ÿ[bð¾64133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236418591 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14706359310336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53689351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24085311 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2104750388878848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109046506 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109046504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29603779 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14545124343296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2104750388878848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475016760729600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚11702.760886772 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127372085 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460310401419264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438éãWe (š+†P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2093974149493961 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14522595198153 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14522935748043 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2104750030500352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24085308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29604040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2093974507083211 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2104750030500352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14545124343296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109046504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2079451554295808 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2090205264535552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14706447887360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2079451571335168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127372085 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127362108 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460310420031488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53689087 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236408612 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2090205264535552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236418591 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14706359310336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53689351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24085311 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2104750388878848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109046506 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109046504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29603779 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14545124343296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2104750388878848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475016760729600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚11702.760886772 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127372085 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460310401419264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ë’Ê4›+/ _wandbruntime‚11703  _runtime‚11703 ´ÿ¥'œ+‹N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29605836 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109054136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460343848790016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2090352488525824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2090352488525824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127371160 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2094122028808917 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2079598064967680 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109054138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127381151 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29606116 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚11703.379434608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53692889 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2090352505565184 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127371160 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2104898284156928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24087053 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2104898642542592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14545795631104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14707371575296 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2079598082007040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1132 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236435289 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236425296 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109054136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13749 % _timestamp‚1.7842845112535362e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14523606251875 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2104898642542592 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109054138 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236435289 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14523946801877 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169} \ÑÎ+N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109054138 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127381151 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29606116 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚11703.379434608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53692889 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2090352505565184 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127371160 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2104898284156928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24087053 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2104898642542592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14545795631104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14707371575296 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2079598082007040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1132 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236435289 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236425296 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109054136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13749 % _timestamp‚1.7842845112535362e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14523606251875 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2104898642542592 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109054138 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_prXH ool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236435289 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14523946801877 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29605836 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109054136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚460343848790016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2090352488525824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2090352488525824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127371160 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2094122028808917 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2079598064967680 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24172561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127826537 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53879989 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127816560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461981839345664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1135 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24172558 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14595819845120 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109428106 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2101350299574469 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14757518786048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14595819845120 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2097567298977792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2086777087889408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14757611557376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2112162777483776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127826537 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53880257 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 237254643 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461981857957888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109428106 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚13799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2112162777483776 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2101350657163719 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842845505384698e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2112163135862272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127816560 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2086777104928768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14573211685061 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2097567316017152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14595478505984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476739358131712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14573552234951 _runtime‚11742.664492266 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476739469515264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29707431 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109428104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2097567298977792 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2097567316017152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888»êðÇ (ª+†P XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14757611557376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2112162777483776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127826537 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53880257 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 237254643 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461981857957888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109428106 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚13799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2112162777483776 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2101350657163719 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7842845505384698e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2112163135862272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127816560 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2086777104928768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14573211685061 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2097567316017152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14595478505984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476739358131712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14573552234951 _runtime‚11742.664492266 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476739469515264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29707431 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109428104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2097567298977792 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2097567316017152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109428104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14595478505984 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2112163135862272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24172561 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127826537 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53879989 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127816560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461981839345664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1135 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24172558 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14595819845120 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109428106 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2101350299574469 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14757518786048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14595819845120 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2097567298977792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2086777087889408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144ï›F4«+/ _wandbruntime‚11743  _runtime‚11743(vùx¬+ŠN ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127825612 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2101498178889425 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127835603 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚462015268104192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127835603 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109435738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24174303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109435738 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.smawÔ²r˜ll_pool.allocated‚29709768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127825612 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2097714557046784 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2097714557046784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109435736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29709482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14596832479232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2097714540007424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53883785 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2112311389526016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚13799 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 $ _timestamp‚1.784284551150877e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237271341 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚11743.276709245 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14596832479232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237271341 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14596491132928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14758541131776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14574222738783 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2112311031140352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237261348 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14758631798784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237261348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127825612 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2101498178889425 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127835603 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚462015268104192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127835603 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109435738 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24174303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109435738 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29709768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127825612 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2097714557046784 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2097714557046784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109435736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29709482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14596832479232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2097714540007424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53883785 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2112311389526016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚13799 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 $ _timestamp‚1.784284551150877e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237271341 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚11743.276709245 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14596832479232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237271341 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14596491132928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14758541131776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14574222738783 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2112311031140352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237261348 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14758631798784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237261348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FË4·+/ _wandbruntime‚11782  _runtime‚11782P*—6̸+Æ ' lr-AdamW/pg1‚0.00010585396697274537 ' lr-AdamW/pg2‚0.00010585396697274537 trainer/global_step‚13849 % _timestamp‚1.7842845904586823e+09 _runtime‚11782.584314656 _step‚1138¿á`µÑ¹+Ë ' lr-AdamW/pg1‚0.00010585396697274537 ' lr-AdamW/pg2‚0.00010585396697274537 trainer/global_step‚13849 % _timestamp‚1.7842845904586823e+09 _runtime‚11782.584314656 _step‚1138ò÷ý[x4º+/ _wandbruntime‚11782  _runtime‚11782˜DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚478462030605312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463653277272064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238090695 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14623828171969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14646174007808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128280989 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29811901 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2094102621483008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2119575882845696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24259808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2094102638522368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14808833521664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2119575524467200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109809704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2104929350459392 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109809706 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚13849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128271012 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2108726807244227 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128280989 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14646515346944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14624168721859 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14646174007808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2108726449654977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109809706 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2104929367498752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109809704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24259811 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128271012 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238080716 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2119575882845696 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2104929367498752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3ˆŒÜ¼+…P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14646174007808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128280989 A8DeviceS÷°øUI'tatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29811901 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2094102621483008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2119575882845696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24259808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2094102638522368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14808833521664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2119575524467200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109809704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2104929350459392 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109809706 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚13849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128271012 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2108726807244227 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128280989 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14646515346944 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14624168721859 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14646174007808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2108726449654977 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109809706 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2104929367498752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109809704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24259811 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128271012 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238080716 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2119575882845696 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2104929367498752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2119575524467200 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚463653295884288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14646515346944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842845904597852e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2104929350459392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚478462030605312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463653277272064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238090695 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14623828171969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888óÅž­‹4½+/ _wandbruntime‚11783  _runtime‚11783%€Œ'¾+‹N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2108873971380571 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238097400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2105076608528384 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2105076608528384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128290055 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238097400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14809860992512 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14625179775693 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238107393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚13849 % _timestamp‚1.7842845910738828e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54075263 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128290055 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2094249149194240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1140 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2105076591489024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14624839225691 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24261553 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238107393 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14647527981056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109817338 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2119723778123776 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109817338 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2119724136509440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54075535 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478496585635328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0JØÿœ'¿+N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128290055 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238097400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14809860992512 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14625179775693 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238107393 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚13849 % _timestamp‚1.7842845910738828e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54075263 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128290055 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2094249149194240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1140 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2105076591489024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14624839225691 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24261553 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238107393 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14647527981056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109817338 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2119723778123776 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109817338 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2119724136509440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54075535 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478496585635328 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2108873971380571 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238097400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2105076608528384 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2105076608528384ô¤)Q4À+/ _wandbruntime‚11783  _runtime‚11783Ã`‚| Á+‡ /loss/FBt2/detection_loss‚0.6715970870784738 0loss/FRt2/assignment_loss‚0.7006451273427426 2loss/SRqqt1/assignment_loss‚0.4455695558392084 2loss/SRqqt2/assignment_loss‚0.4277173217083748 /loss/FRt2/detection_loss‚1.2592334481626433 1loss/SRqqt1/detection_loss‚0.8721906093362669 epoch‚34 _runtime‚11783.20261214 0loss/FRt1/assignment_loss‚1.5225928731919374 /loss/FBt1/detection_loss‚0.3631792833339518 trainer/global_step‚13849 _step‚1141 2loss/FBt2/assignment_loss‚0.000196908788351183 1loss/SRqqt2/detection_loss‚0.5950353086256438 &loss/total_loss‚7.4032295359416995 % _timestamp‚1.7842845910772855e+09 2loss/FBt1/assignment_loss‚0.008026122133704645 /loss/FRt1/detection_loss‚0.5372458904004006´eä_Â+Œ &loss/total_loss‚7.4032295359416995 % _timestamp‚1.7842845910772855e+09 2loss/FBt1/assignment_loss‚0.008026122133704645 /loss/FRt1/detection_loss‚0.5372458904004006 /loss/FBt2/detection_loss‚0.6715970870784738 0loss/FRt2/assignment_loss‚0.7006451273427426 2loss/SRqqt1/assignment_loss‚0.4455695558392084 2loss/SRqqt2/assignment_loss‚0.4277173217083748 /loss/FRt2/detection_loss‚1.2592334481626433 1loss/SRqqt1/detection_loss‚0.8721906093362669 epoch‚34 _runtime‚11783.20261214 0loss/FRt1/assignment_loss‚1.5225928731919374 /loss/FBt1/detection_loss‚0.3631792833339518 trainer/global_step‚13849 _step‚1141 2loss/FBt2/assignment_loss‚0.000196908788351183 1loss/SRqqt2/detection_loss‚0.5950353086256438õ—À\‘Ã+‚(:† µ‹èÒŒšŽ£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Óü9Ä+‚(:ô µ‹èÒû˜Ù¥ disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 482.56640625 disk.rbd3.out‚0$ memory_percent‚97.36382961273193 cpu‚26.029685943562278( disk.nvme0c0n1.out‚53403.86181640625 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚113.63720703125 disk.dm-0.in‚ 144864.8125 proc.memory.rssMB‚ 2822.28125 disk.sda1.out‚0 disk.sda.out‚ 482.56640625) proc.memory.percent‚17.22583770751953) disk./.usagePercent‚36.82928355697606 disk.rbd0.out‚0UÖ-ÞÆ disk.rbd1.in‚0.0625 disk.rbd3.in‚0( proc.memory.availableMB‚ 431.91015625& disk.nvme0n1.out‚53403.86181640625 disk.nvme0n1.in‚ 144864.8125! disk.nvme0c0n1.in‚ 144864.8125 disk.sda2.out‚0 network.sent‚ 2.1991466e+07 network.recv‚ 3.147883e+06 disk.sda3.out‚ 482.56640625# disk.dm-0.out‚53403.86181640625 proc.cpu.threads‚17$ disk./.usageGB‚548.7829399108887 disk.drbd2165.out‚0 disk.rbd0.in‚0JJ×Z“Å+‚(:ˆ Ä‹èÒ‰†¤ gpu.0.gpu‚67.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑW¿k Æ+‚(:þ Ä‹èÒÙÙÊ¥ disk.drbd2165.in‚0$ memory_percent‚97.36649990081787) proc.memory.percent‚17.22583770751953( disk.nvme0c0n1.out‚53411.05517578125 disk.dm-1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚53411.05517578125 disk.sda1.out‚0$ disk./.usageGB‚548.7832565307617! disk.rbd1.out‚113.69970703125 disk.dm-0.in‚144865.52734375 proc.cpu.threads‚17 disk.sda3.out‚ 482.9296875 network.recv‚ 3.152036e+06% disk.nvme0c0n1.in‚144865.52734375 disk.sda2.in‚0 disk.drbd2165.out‚0 cpu‚26.029290193187176 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 431.47265625 proc.memory.rssMB‚ 2822.28125 network.sent‚ 2.2054459e+07 disk.sda.out‚ 482.9296875# disk.nvme0n1.in‚144865.52734375 disk.dm-1.out‚ 482.9296875 disk.rbd1.in‚0.0625# disk.dm-0.out‚53411.05517578125* disk./.usagePercent‚36.829304805601396 disk.rbd3.out‚0 disk.sda1.in‚0!ÿnžÇ+‚(:“ Ó‹èÒëÐÅ¢ gpu.0.gpu‚66.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ºU_þÈ+‚(:ó Ó‹èÒΩõ¤% disk.nvme0c0n1.in‚144866.35546875% disk.nvme0c0n1.out‚53420.01953125 disk.dm-0.out‚53420.01953125$ memory_percent‚97.36804962158203 proc.memory.rssMB‚ 2822.28125) proc.memory.percent‚17.22583770751953 network.recv‚ 3.155337e+06$ disk./.usageGB‚548.7838020324707 disk.sda2.out‚0 disk.sda3.out‚ 483.30078125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚144866.35546875 proc.cpu.threads‚17 network.sent‚ 2.2059462e+07 disk.rbd0.in‚0 disk.sda.in‚0 cpu‚26.02891911650618) disk./.usagePercent‚36.82934141467879# disk.nvme0n1.out‚53420.01953125 disk.dm-1.in‚0% proc.memory.availableMB‚ 431.21875 disk.dm-1.out‚ 483.30078125! disk.rbd1.out‚113.69970703125 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda.out‚ 483.30078125 disk.rbd3.out‚0# disk.nvme0n1.in‚144866.35546875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda3.in‚0]q„4É+/ _wandbruntime‚11822  _runtime‚11822Š¥DïÉÊ+à _step‚1142 & lr-AdamW/pg1‚0.0001042789619911259 & lr-AdamW/pg2‚0.0001042789619911259 trainer/global_step‚13899 $ _timestamp‚1.784284630333076e+09 _runtime‚11822.458641565IÝ \ÎË+È trainer/global_step‚13899 $ _timestamp‚1.784284630333076e+09 _runtime‚11822.458641565 _step‚1142 & lr-AdamW/pg1‚0.0001042789619911259 & lr-AdamW/pg2‚0.0001042789619911259ö@"Ð4Ì+/ _wandbruntime‚11822  _runtime‚11822óýGÜ(Í+P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14860087082496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚13899 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14696869509632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110191304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110191306 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14860004796928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2126988271450624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54262412 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480184820893184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24347058 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29915351 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14674444658877 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚1143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238926747 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29915101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480184719995392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24347061 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14696869509632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2ò¼aT (Î+†P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24347061 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14696869509632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14860087082496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚13899 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14696869509632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110191304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110191306 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14860004796928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2126988271450624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54262412 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480184820893184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24347058 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29915351 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14674444658877 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚1143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238926747 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29915101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480184719995392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136÷üáͽ4Ï+/ _wandbruntime‚11823  _runtime‚11823^A´a Ð+‹N FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480219280792576 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110198936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128744507 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2116250121461079 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465358143956992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110198938 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110198936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110198938 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2116250479050441 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2101574682787840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238943445 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14861031750656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14698223482880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24348803 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2127136525107200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14698223482880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2101574665748480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14675455712599 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2112438660009984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238933452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚11823.069266007 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238943445 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29917136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480219175707648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2112438642970624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2127136883492864 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54265939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2112438660009984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14675796262601 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚13899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238933452 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14697882136576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128734516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24348806 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 % _timestamp‚1.7842846309434655e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376è¼y–'Ñ+N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480219280792576 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110198936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128744507 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2116250121461079 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465358143956992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110198938 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110198936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110198938 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2116250479050441 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2101574682787840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238943445 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14861031750656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14698223482880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24348803 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2127136525107200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14698223482880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2101574665748480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14675455712599 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2112438660009984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238933452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚11823.069266007 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238943445 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29917136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480219175707648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2112438642970624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2127136883492864 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54265939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2112438660009984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14675796262601 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚13899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238933452 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14697882136576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128734516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24348806 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 % _timestamp‚1.7842846309434655e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14747906350592 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14725401695675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110572904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129189893 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54452999 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2123479107405243 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚466996171737088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2108753705709568 trainer/global_step‚13949 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2134401018434048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129179916 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2119653453422592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14747565011456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129189893 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2134401018434048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24434308 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110572904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2134401376812544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14747906350592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481907434176000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129179916 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976P)/=Þ+†P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129179916 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2119653453422592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14747565011456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129189893 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2134401018434048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24434308 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110572904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2134401376812544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14747906350592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481907434176000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129179916 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2134401376812544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2119653470461952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2108753688670208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14725061145785 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2119653453422592 % _timestamp‚1.7842846700539486e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110572906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30018691 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14747565011456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚11862.179758925 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2123478749815993 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239752820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30018954 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239762799 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14911169667584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1147 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14747906350592 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14725401695675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110572904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129189893 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54452999 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2123479107405243 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚466996171737088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2108753705709568 trainer/global_step‚13949 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2134401018434048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169û )x4ß+/ _wandbruntime‚11862  _runtime‚11862înÔß'à+ŠN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129198959 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239779497 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2119800711491584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14726072199507 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467029581883392 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129198959 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467029600495616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14912284630016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2134549272090624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14748577638400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110580538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110580538 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54457065 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54456793 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30021009 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14912191865856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2119800711491584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2134549630476288 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110580536 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2119800694452224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784284670669441e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚11862.795350205 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24436053 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2123626271541587 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2108900216381440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2119800694452224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14726412749509 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239769504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2108900199342080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239779497 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2123626629130949 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239779497 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2123626629130949 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14726072199507 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467029581883392 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129198959 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467029600495616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14912284630016 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2134549272090624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14748577638400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110580538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110580538 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54457065 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54456793 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30021009 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14912191865856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2119800711491584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2134549630476288 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110580536 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2119800694452224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784284670669441e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚11862.795350205 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24436053 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2123626271541587 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2108900216381440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438ü†µPõ4â+/ _wandbruntime‚11862  _runtime‚11862,€™ýã+‰ 3loss/SRqqt2/assignment_loss‚0.41475339407782136 2loss/FBt2/assignment_loss‚0.009018357351642463 /loss/FBt2/detection_loss‚0.4928664093454098 &loss/total_loss‚7.0566625358847945 /loss/FRt1/detection_loss‚0.4357929027081231 0loss/FBt1/detection_loss‚0.37684211071714374 epoch‚34 % _timestamp‚1.7842846706714756e+09 0loss/FRt2/assignment_loss‚0.8340569572628669 3loss/SRqqt1/assignment_loss‚0.33546693735294425 1loss/SRqqt1/detection_loss‚0.7424441387544798 trainer/global_step‚13949 _runtime‚11862.79679878 _step‚1149 0loss/FRt1/assignment_loss‚1.6063692552182771 1loss/FBt1/assignment_loss‚0.02068298894108917 /loss/FRt2/detection_loss‚1.2338653163876239 1loss/SRqqt2/detection_loss‚0.5545037677673729˜Ó6•ä+Ž 0loss/FRt1/assignment_loss‚1.6063692552182771 1loss/FBt1/assignment_loss‚0.02068298894108917 /loss/FRt2/detection_loss‚1.2338653163876239 1loss/SRqqt2/detection_loss‚0.5545037677673729 3loss/SRqqt2/assignment_loss‚0.41475339407782136 2loss/FBt2/assignment_loss‚0.009018357351642463 /loss/FBt2/detection_loss‚0.4928664093454098 &loss/total_loss‚7.0566625358847945 /loss/FRt1/detection_loss‚0.4357929027081231 0loss/FBt1/detection_loss‚0.37684211071714374 epoch‚34 % _timestamp‚1.7842846706714756e+09 0loss/FRt2/assignment_loss‚0.8340569572628669 3loss/SRqqt1/assignment_loss‚0.33546693735294425 1loss/SRqqt1/detection_loss‚0.7424441387544798 trainer/global_step‚13949 _runtime‚11862.79679878 _step‚1149ý^ã­‘å+‚(:† €ŒèÒØÞ©£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.457' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä_8æ+‚(:ö €ŒèÒÄ˾¤ disk.sda1.out‚0$ disk./.usageGB‚548.7837295532227 disk.dm-1.out‚ 484.3203125" disk.nvme0n1.in‚144867.8984375( proc.memory.availableMB‚ 431.45703125 proc.memory.rssMB‚ 2822.28125 network.recv‚ 3.16654e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0) proc.memory.percent‚17.22583770751953 disk.dm-1.in‚0% disk.nvme0n1.out‚53439.0048828125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚53439.0048828125 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚144867.8984375 cpu‚26.027720336547407) disk./.usagePercent‚36.82933655053564 disk.sda2.in‚0 disk.rbd1.in‚0.0625" disk.dm-0.out‚53439.0048828125 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 484.3203125 disk.dm-0.in‚144867.8984375 disk.drbd2165.in‚0 disk.sda.out‚ 484.3203125$ memory_percent‚97.36659526824951! disk.rbd1.out‚113.82470703125 network.sent‚ 2.2132633e+073*Ä‘ç+‚(:† ŒèÒ­”­¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 õäè+‚(:û ŒèÒòóФ disk.rbd3.in‚0 network.recv‚ 3.170971e+06 disk.sda3.in‚0! disk.rbd1.out‚113.82470703125 disk.sda1.in‚0% disk.nvme0c0n1.in‚144868.61328125' disk.nvme0c0n1.out‚53448.0869140625 disk.sda2.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚548.7837219238281 disk.rbd1.in‚0.0625 proc.memory.rssMB‚ 2822.28125 network.sent‚ 2.2195674e+07) disk./.usagePercent‚36.82933603852057 disk.dm-1.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚144868.61328125% disk.nvme0n1.out‚53448.0869140625 disk.dm-0.in‚144868.61328125" disk.dm-0.out‚53448.0869140625 disk.drbd2165.out‚0 disk.sda.out‚ 484.64453125 disk.sda2.out‚0$ memory_percent‚97.36676216125488 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 431.4296875 disk.sda3.out‚ 484.64453125) proc.memory.percent‚17.22583770751953 cpu‚26.02732844530441 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.out‚ 484.64453125î$>žé+‚(:“ žŒèÒð°°¢ gpu.0.gpu‚75.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Smãëê+‚(:ö žŒèÒá‹ß¤ disk.sda2.in‚0$ memory_percent‚97.36676216125488 proc.cpu.threads‚17 network.sent‚ 2.2200679e+07 disk.rbd0.in‚0 disk.dm-0.in‚144869.44140625 cpu‚26.02693766888166 disk.sda.out‚ 484.96875 disk.dm-1.out‚ 484.96875* disk./.usagePercent‚36.829336806543175# disk.nvme0n1.in‚144869.44140625% disk.nvme0c0n1.in‚144869.44140625' proc.memory.availableMB‚ 431.4296875 network.recv‚ 3.173999e+06$ disk./.usageGB‚548.7837333679199& disk.nvme0n1.out‚53455.26318359375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚113.88720703125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 484.96875 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚53455.26318359375) proc.memory.percent‚17.22583770751953( disk.nvme0c0n1.out‚53455.26318359375 disk.sda.in‚0 disk.rbd1.in‚0.0625 proc.memory.rssMB‚ 2822.28125 disk.drbd2165.in‚0-¸6f4ë+/ _wandbruntime‚11902  _runtime‚11902ã„Ìì+Æ ' lr-AdamW/pg1‚0.00010115449760834202 ' lr-AdamW/pg2‚0.00010115449760834202 trainer/global_step‚13999 % _timestamp‚1.7842847100085227e+09 _runtime‚11902.134107488 _step‚1150î`XÑí+Ë ' lr-AdamW/pg1‚0.00010115449760834202 ' lr-AdamW/pg2‚0.00010115449760834202 trainer/global_step‚13999 % _timestamp‚1.7842847100085227e+09 _runtime‚11902.134107488 _step‚1150þp‰24î+/ _wandbruntime‚11902  _runtime‚11902|e6ï+P trainer/global_step‚13999 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2127015504904192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2127015504904192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468667609663488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7842847100096707e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110954506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14962355726848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2141814123795968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.oüTÉ Ñ!n_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14776018182583 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110954504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24521561 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240588872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129644345 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14798260513280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚483630058161664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚468667591051264 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2130854899896501 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2141814123795968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129644345 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2141813765417472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110954506 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2130855257485751 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14775677632693 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14798260513280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54644093 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2141813765417472 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110954504 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129634368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2127015521943552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚483629946778112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚1151 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2127015521943552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14798601852416 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚11902.135469879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14962448498176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24521558 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24521561 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240588872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129644345 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14798260513280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚483630058161664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚468667591051264 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2130854899896501 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2141814123795968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129644345 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2141813765417472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110954506 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2130855257485751 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14775677632693 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14798260513280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54644093 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2141813765417472 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110954504 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129634368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2127015521943552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚483629946778112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚1151 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2127015521943552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14798601852416 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚11902.135469879 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14962448498176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24521558 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11902.749147961 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2141962377459712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240605556 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110962138 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2127162745933824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚468701019809792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110962136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14776688686415 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468701038422016 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240615549 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2116225749975040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2141962377459712 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2127162762973184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129653411 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483664398458368 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2131002421622095 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483664505640448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14963467218432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14777029236417 _step‚1152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110962138 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2131002779211457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240615549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24523306 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129653411 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129643420 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2127162762973184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54647887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚13999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129643420 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54648135 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2116225732935680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14799273140224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842847106233256e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468701038422016 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240615549 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2116225749975040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2141962377459712 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2127162762973184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129653411 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483664398458368 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2131002421622095 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483664505640448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14963467218432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14777029236417 _step‚1152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110962138 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2131002779211457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240615549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24523306 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129653411 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.c!+¸³[urrent‚248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129643420 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2127162762973184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54647887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚13999 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129643420 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54648135 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2116225732935680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14799273140224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842847106233256e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11902.749147961 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2141962377459712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240605556 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110962138 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2127162745933824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚468701019809792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110962136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14776688686415 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888€ 1ê«4ô+/ _wandbruntime‚11902  _runtime‚11902`Dt' õ+… %loss/total_loss‚7.211477831724188 2loss/SRqqt2/assignment_loss‚0.4416155176461233 0loss/SRqqt2/detection_loss‚0.623534780513848 0loss/FBt1/detection_loss‚0.37612897365195547 _step‚1153 2loss/FBt2/assignment_loss‚0.000378886780134893 .loss/FRt1/detection_loss‚0.510385922608313 trainer/global_step‚13999 /loss/FBt2/detection_loss‚0.6241738371878724 epoch‚34 0loss/FRt1/assignment_loss‚1.4640373468399048 0loss/FRt2/assignment_loss‚0.7486599932306062 2loss/FBt1/assignment_loss‚0.002378197368136122 /loss/FRt2/detection_loss‚1.2022927461208193 0loss/SRqqt1/detection_loss‚0.814818710964744 2loss/SRqqt1/assignment_loss‚0.4030729188117304 % _timestamp‚1.7842847106274452e+09 _runtime‚11902.753107015£$R‡ö+Š trainer/global_step‚13999 /loss/FBt2/detection_loss‚0.6241738371878724 epoch‚34 0loss/FRt1/assignment_loss‚1.4640373468399048 0loss/FRt2/assignment_loss‚0.7486599932306062 2loss/FBt1/assignment_loss‚0.002378197368136122 /loss/FRt2/detection_loss‚1.2022927461208193 0loss/SRqqt1/detection_loss‚0.814818710964744 2loss/SRqqt1/assignment_loss‚0.4030729188117304 % _timestamp‚1.7842847106274452e+09 _runtime‚11902.753107015 %loss/total_loss‚7.211477831724188 2loss/SRqqt2/assignment_loss‚0.4416155176461233 0loss/SRqqt2/detection_loss‚0.623534780513848 0loss/FBt1/detection_loss‚0.37612897365195547 _step‚1153 2loss/FBt2/assignment_loss‚0.000378886780134893 .loss/FRt1/detection_loss‚0.510385922608313 Kó~“÷+‚(:ˆ ­ŒèÒ¸›´£ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚283.649' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£­.êêø+‚(:ß ­ŒèÒŽ‚€¥$ disk.nvme0c0n1.out‚ 53463.7734375 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 144870.15625( proc.memory.availableMB‚ 431.16015625 disk.dm-1.out‚485.375 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.82930301354866 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.out‚ 53463.7734375" disk.nvme0n1.out‚ 53463.7734375# proc.memory.rssMB‚ 2822.28515625$ disk./.usageGB‚548.7832298278809 disk.rbd3.in‚0 disk.dm-0.in‚ 144870.15625) proc.memory.percent‚17.22586154937744 network.sent‚ 2.2205794e+07 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 144870.15625 disk.sda3.out‚485.375 disk.dm-1.in‚0! disk.rbd1.out‚113.88720703125 disk.rbd3.out‚0 cpu‚26.028951965221196$ memory_percent‚97.36840724945068 network.recv‚ 3.177696e+06 disk.sda.out‚485.375 disk.sda2.out‚0 disk.rbd0.out‚0-_Ãû4ù+/ _wandbruntime‚11919  _runtime‚11919Ü”òK§=ú+¡{ :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21845214068889618 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7986067533493042 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9298759698867798 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.048084814101457596 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6288325190544128 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7226277589797974 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5895625352859497 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 6validation_average_jet_accuracy‚0.3865242898464203 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416865348816 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.06481482088565826 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16149260103702545 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 trainer/global_step‚13999 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7685609459877014 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08680955320596695 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6649632453918457 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_5_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8038548827171326 *jet/accuracy_1_of_4‚0.9830800294876099 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8664526343345642 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9232513904571533 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8484684824943542 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22696688771247864 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416666328907013 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2450736165046692 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2517016530036926 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868807196617126 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2626422345638275 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24047735333442688 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.99151211977005 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8037226796150208 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3440511226654053 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4991645812988281 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 *jet/accuracy_1_of_1‚0.3461182415485382 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7132317423820496 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8096750378608704 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37297844886779785 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949916005134583 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6203892230987549 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9808968305587769 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.574580192565918 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 'particle/f_score‚0.6318797469139099 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6784510612487793 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6196001172065735 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8997836112976074 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19798578321933746 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6889058351516724 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3730222284793854 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7751784324645996 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18346746265888214 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6419162750244141 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3002128005027771 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2806892991065979 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02132382057607174 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_2‚0.7990126609802246 _step‚1154 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2896447777748108 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9916321635246277 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021793683990836143 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6263120770454407 particle/accuracy_1_of_3‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022751515731215477 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23432543873786926 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6643055081367493 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022092126309871674 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2476702183485031 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.349778413772583 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923775792121887 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975825548171997 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.06481482088565826 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6534109711647034 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6550623178482056 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6472479701042175 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49254679679870605 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2995588183403015 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9904484748840332 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28943341970443726 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28943341970443726 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6876230835914612 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.867475152015686 /particle/accuracy_4_of_4‚0.9581526517868042 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6205832362174988 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *particle/specificity‚0.905103325843811 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7969387769699097 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (particle/accuracy‚0.8443428874015808 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6433302760124207 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.011098379269242287 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962319135665894 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7265117764472961 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2525652050971985 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.867475152015686 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31682419776916504 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8075208067893982 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23892518877983093 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2648859918117523 particle/accuracy_1_of_5‚1 )loss/val_total_loss‚7.943910731101338 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20768222212791443 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5497996211051941 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6414779424667358 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04540162906050682 +particle/sensitivity‚0.6221582293510437 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19332405924797058 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6330538988113403 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7749254107475281 _runtime‚11919.843501872 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2906772792339325 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6834593415260315 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018546059727668762 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983811378479004 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3059397339820862 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018546059727668762 particle/accuracy_2_of_4‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5008963346481323 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5827906727790833 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3873628079891205 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6288325190544128 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7276850342750549 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.051629021763801575 epoch‚34 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6345601081848145 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9954454302787781 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7158493995666504 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7924752235412598 +jet/accuracy_3_of_3‚0.06378723680973053 particle/accuracy_3_of_3‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6902260780334473 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968428611755371 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 )jet/accuracy_2_of_4‚0.792914092540741 +jet/accuracy_3_of_4‚0.08499640971422195 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5249999761581421 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9282920360565186 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2993037700653076 % _timestamp‚1.7842847277172012e+09 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5641339421272278 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6090472936630249 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09890514612197876 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_4_of_5‚1 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9956493377685547 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6893779039382935 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9874330163002014 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.730751633644104 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7747677564620972 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6818717122077942 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.533994734287262 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26449504494667053 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25384393334388733 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46522316336631775 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039537057280540466.\¿û+¦{ 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2517016530036926 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868807196617126 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2626422345638275 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24047735333442688 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.99151211977005 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8037226796150208 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3440511226654053 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4991645812988281 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 *jet/accuracy_1_of_1‚0.3461182415485382 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7132317423820496 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8096750378608704 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37297844886779785 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949916005134583 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6203892230987549 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9808968305587769 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.574580192565918 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 'particle/f_score‚0.6318797469139099 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6784510612487793 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6196001172065735 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8997836112976074 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19798578321933746 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6889058351516724 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3730222284793854 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7751784324645996 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18346746265888214 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6419162750244141 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3002128005027771 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2806892991065979 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02132382057607174 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_2‚0.7990126609802246 _step‚1154 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2896447777748108 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9916321635246277 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021793683990836143 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6263120770454407 particle/accuracy_1_of_3‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022751515731215477 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23432543873786926 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6643055081367493 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022092126309871674 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2476702183485031 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.349778413772583 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923775792121887 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975825548171997 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.06481482088565826 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6534109711647034 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6550623178482056 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6472479701042175 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49254679679870605 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2995588183403015 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9904484748840332 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28943341970443726 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28943341970443726 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6876230835914612 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.867475152015686 /particle/accuracy_4_of_4‚0.9581526517868042 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6205832362174988 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *particle/specificity‚0.905103325843811 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7969387769699097 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (particle/accuracy‚0.8443428874015808 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6433302760124207 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.011098379269242287 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9962319135665894 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7265117764472961 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2525652050971985 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.867475152015686 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31682419776916504 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8075208067893982 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23892518877983093 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2648859918117523 particle/accuracy_1_of_5‚1 )loss/val_total_loss‚7.943910731101338 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20768222212791443 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5497996211051941 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6414779424667358 @&Purity/1FRt1SRqqt1FBt/eSÝ…‡ívent_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04540162906050682 +particle/sensitivity‚0.6221582293510437 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19332405924797058 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6330538988113403 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7749254107475281 _runtime‚11919.843501872 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2906772792339325 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6834593415260315 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018546059727668762 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983811378479004 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3059397339820862 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018546059727668762 particle/accuracy_2_of_4‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5008963346481323 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5827906727790833 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3873628079891205 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6288325190544128 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7276850342750549 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.051629021763801575 epoch‚34 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6345601081848145 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9954454302787781 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7158493995666504 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7924752235412598 +jet/accuracy_3_of_3‚0.06378723680973053 particle/accuracy_3_of_3‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6902260780334473 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968428611755371 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 )jet/accuracy_2_of_4‚0.792914092540741 +jet/accuracy_3_of_4‚0.08499640971422195 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5249999761581421 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9282920360565186 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2993037700653076 % _timestamp‚1.7842847277172012e+09 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5641339421272278 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6090472936630249 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09890514612197876 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_4_of_5‚1 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9956493377685547 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6893779039382935 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9874330163002014 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.730751633644104 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7747677564620972 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6818717122077942 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.533994734287262 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26449504494667053 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25384393334388733 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46522316336631775 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039537057280540466 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21845214068889618 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7986067533493042 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9298759698867798 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.048084814101457596 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6288325190544128 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7226277589797974 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5895625352859497 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 6validation_average_jet_accuracy‚0.3865242898464203 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416865348816 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.06481482088565826 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16149260103702545 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 trainer/global_step‚13999 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7685609459877014 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08680955320596695 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6649632453918457 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_5_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8038548827171326 *jet/accuracy_1_of_4‚0.9830800294876099 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8664526343345642 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9232513904571533 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8484684824943542 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22696688771247864 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416666328907013 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2450736165046692 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272‚ q£4¹žü+‚(:“ ¼ŒèÒ°Ö°£ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚259.109' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.821799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Ü¿µðý+‚(:å ¼ŒèҤ쀥 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-0.out‚ 53467.4375! disk.nvme0c0n1.out‚ 53467.4375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0$ memory_percent‚97.36855030059814( proc.memory.availableMB‚ 431.13671875) proc.memory.percent‚17.22586154937744 network.sent‚ 2.2268955e+07$ disk./.usageGB‚548.7832946777344 disk.sda.out‚ 485.65234375) disk./.usagePercent‚36.82930736567674 disk.nvme0n1.out‚ 53467.4375 disk.rbd0.out‚0 cpu‚26.031818620296654 disk.sda3.out‚ 485.65234375! disk.rbd1.out‚113.94970703125" disk.nvme0c0n1.in‚ 144870.15625 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 144870.15625# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17 network.recv‚ 3.182523e+06 disk.nvme0n1.in‚ 144870.15625 disk.dm-1.out‚ 485.65234375À—ôæ“þ+‚(:ˆ ËŒèÒáݹ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04A.¶ÿ+‚(:ø ËŒèÒ†ÏÞ¤ disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd1.out‚113.94970703125! disk.nvme0n1.in‚ 144870.984375) disk./.usagePercent‚36.82929763739043$ disk./.usageGB‚548.7831497192383( disk.nvme0c0n1.out‚53478.75244140625 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 485.9609375 disk.rbd3.in‚0# disk.dm-0.out‚53478.75244140625 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0$ memory_percent‚97.36826419830322 cpu‚26.031446277222255 network.sent‚ 2.23264e+07 network.recv‚ 3.186545e+06 disk.sda3.out‚ 485.9609375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0.0625 proc.cpu.threads‚17 disk.sda.in‚0) proc.memory.percent‚17.22586154937744& disk.nvme0n1.out‚53478.75244140625# proc.memory.rssMB‚ 2822.28515625# disk.nvme0c0n1.in‚ 144870.984375 disk.sda.out‚ 485.9609375 disk.dm-1.in‚0 disk.dm-0.in‚ 144870.984375 disk.sda1.out‚0( proc.memory.availableMB‚ 431.18359375>HšQ€,‚(:’ ÚŒèÒ’À¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.219' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.64379999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ÆÀ,‚(:ú ÚŒèÒÞ‹Ñ¥ disk.rbd3.in‚0$ proc.memory.availableMB‚431.1875 proc.cpu.threads‚17! disk.rbd1.out‚114.07470703125# disk.nvme0n1.in‚144871.69921875 disk.sda1.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.in‚144871.69921875 disk.dm-1.out‚ 486.2109375 cpu‚26.03100711070675 disk.rbd0.out‚0( disk.nvme0c0n1.out‚53485.54931640625 disk.sda.in‚0 network.sent‚ 2.2333487e+07( disk./.usagePercent‚36.8293071096692 disk.rbd3.out‚0 disk.dm-0.in‚144871.69921875& disk.nvme0n1.out‚53485.54931640625 disk.drbd2165.in‚0 disk.sda.out‚ 486.2109375 disk.sda3.out‚ 486.2109375 disk.dm-1.in‚0$ memory_percent‚97.36824035644531# proc.memory.rssMB‚ 2822.28515625$ disk./.usageGB‚548.7832908630371 disk.sda2.out‚0) proc.memory.percent‚17.22586154937744 network.recv‚ 3.189597e+06 disk.rbd1.in‚0.0625# disk.dm-0.out‚53485.54931640625 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0´ÜÇâ4‚,/ _wandbruntime‚11959  _runtime‚11959ÊRÿkʃ,Ä & lr-AdamW/pg1‚9.960523888568641e-05 & lr-AdamW/pg2‚9.960523888568641e-05 trainer/global_step‚14049 % _timestamp‚1.7842847670649023e+09 _runtime‚11959.190497255 _step‚1155‡(Ï„,É _step‚1155 & lr-AdamW/pg1‚9.960523888568641e-05 & lr-AdamW/pg2‚9.960523888568641e-05 trainer/global_step‚14049 % _timestamp‚1.7842847670649023e+09 _runtime‚11959.190497255ƒ Ž ìù4…,/ _wandbruntime‚11959  _runtime‚11959ñ‹5Œ(†,P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241625891 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130159142 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2140157432916318 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2136307140075520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485386809235456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚11959.191757915 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241615912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14866689220608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 % _timestamp‚1.7842847670659437e+09 _step‚1156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30260363 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14844354394720 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15031849545728 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2140157790505568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2151174170635264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130169119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚14049 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24611328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14867030559744 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2136307123036160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14867030559744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14866689220608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130159142 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30260621 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111456770 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24611331 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111456772 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2136307123036160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111456772 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111456770 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2151173812256768 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2136307140075520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470354941077504 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2125313436110848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54871952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2151173812256768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485386706240512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15031765163008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470354959689728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2151174170635264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2125313419071488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14844013844830 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130169119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2&™³ê$‡,†P JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14867030559744 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2136307123036160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14867030559744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14866689220608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130159142 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30260621 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111456770 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24611331 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111456772 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2136307123036160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111456772 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111456770 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2151173812256768 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2136307140075520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470354941077504 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2125313436110848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54871952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2151173812256768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485386706240512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15031765163008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470354959689728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2151174170635264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2125313419071488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14844013844830 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130169119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241625891 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130159142 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2140157432916318 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2136307140075520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485386809235456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚11959.191757915 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241615912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14866689220608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 % _timestamp‚1.7842847670659437e+09 _step‚1156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30260363 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14844354394720 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15031849545728 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2140157790505568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2151 {\"174170635264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130169119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚14049 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24611328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438„ ¾‰J4ˆ,/ _wandbruntime‚11959  _runtime‚11959€n'A'‰,ŠN SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30262447 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130178185 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2136454364065792 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54875520 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2125459929743360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241642589 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14868043193856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2136454381105152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30262704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14845365448554 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241642589 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241632596 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130168194 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2136454364065792 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2136454381105152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚11959.802478487 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14867701847552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111464402 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111464402 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2151322065913344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚485421269122560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130168194 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14845024898552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54875520 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2125459929743360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241642589 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14868043193856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2136454381105152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30262704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14845365448554 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241642589 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241632596 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130168194 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2136454364065792 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2136454381105152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚11959.802478487 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14867701847552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111464402 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111464402 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2151322065913344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚485421269122560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130168194 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14845024898552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130178185 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0… ú¬»Ï4‹,/ _wandbruntime‚11959  _runtime‚11959+4 Œ,… _runtime‚11959.805791029 _step‚1158 /loss/FRt2/detection_loss‚1.1713028586423748 /loss/FBt1/detection_loss‚0.4445620275701423 epoch‚35 0loss/SRqqt2/detection_loss‚0.631939015172417 $loss/total_loss‚7.42240577133769 trainer/global_step‚14049 .loss/FRt2/assignment_loss‚0.73091689117626 2loss/SRqqt1/assignment_loss‚0.3894936898217526 2loss/SRqqt2/assignment_loss‚0.4358037568486019 3loss/FBt1/assignment_loss‚0.0005857997273048673 1loss/SRqqt1/detection_loss‚0.8592333371275226 /loss/FBt2/detection_loss‚0.7632214074514599 % _timestamp‚1.7842847676804597e+09 /loss/FRt1/assignment_loss‚1.485444958104613 4loss/FBt2/assignment_loss‚0.00018409923325153672 /loss/FRt1/detection_loss‚0.5097179304619887®?á,Š /loss/FRt2/detection_loss‚1.1713028586423748 /loss/FBt1/detection_loss‚0.4445620275701423 epoch‚35 0loss/SRqqt2/detection_loss‚0.631939015172417 $loss/total_loss‚7.42240577133769 trainer/global_step‚14049 .loss/FRt2/assignment_loss‚0.73091689117626 2loss/SRqqt1/assignment_loss‚0.3894936898217526 2loss/SRqqt2/assignment_loss‚0.4358037568486019 3loss/FBt1/assignment_loss‚0.0005857997273048673 1loss/SRqqt1/detection_loss‚0.8592333371275226 /loss/FBt2/detection_loss‚0.7632214074514599 % _timestamp‚1.7842847676804597e+09 /loss/FRt1/assignment_loss‚1.485444958104613 4loss/FBt2/assignment_loss‚0.00018409923325153672 /loss/FRt1/detection_loss‚0.5097179304619887 _runtime‚11959.805791029 _step‚1158† 0ÂsžŽ,‚(:“ éŒèÒ·û㢠gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05W,‚(:ý éŒèÒºûठdisk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚53495.181640625) proc.memory.percent‚17.22586154937744 disk.dm-0.in‚144872.52734375( proc.memory.availableMB‚ 431.19921875 cpu‚26.030618638419313% disk.nvme0c0n1.in‚144872.52734375 disk.sda.in‚0 disk.dm-1.out‚ 486.41796875 disk.rbd1.in‚0.0625$ memory_percent‚97.36816883087158# proc.memory.rssMB‚ 2822.28515625# disk.nvme0n1.in‚144872.52734375 disk.sda1.in‚0& disk.nvme0c0n1.out‚53495.181640625 disk.drbd2165.out‚0 disk.sda.out‚ 486.41796875 disk.sda3.out‚ 486.41796875$ disk./.usageGB‚548.7780075073242 disk.drbd2165.in‚0 disk.sda2.out‚0 network.sent‚ 2.2336451e+07 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.recv‚ 3.193161e+06 disk.rbd0.out‚0! disk.rbd1.out‚114.07470703125 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚36.82895253923428$ disk.nvme0n1.out‚53495.1816406255b‰V,‚(:’ øŒèÒÒÖ¬¢ gpu.0.gpu‚36.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.047' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.60940000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0. ”g‘,‚(:ý øŒèÒØ­â¤! disk.rbd1.out‚114.13720703125( proc.memory.availableMB‚ 431.19921875 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚144873.2421875 disk.sda2.out‚0 disk.dm-0.in‚144873.2421875 cpu‚26.030247923559283 disk.sda.out‚ 486.63671875 disk.dm-1.out‚ 486.63671875 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚53501.6708984375) proc.memory.percent‚17.22586154937744 network.recv‚ 3.197228e+06) disk./.usagePercent‚36.82895356326441 disk.rbd1.in‚0.0625 disk.sda3.in‚0# proc.memory.rssMB‚ 2822.28515625$ disk./.usageGB‚548.7780227661133% disk.nvme0n1.out‚53501.6708984375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚53501.6708984375 network.sent‚ 2.2399516e+07" disk.nvme0n1.in‚144873.2421875 disk.rbd0.in‚0 disk.sda.in‚0$ memory_percent‚97.36816883087158 disk.sda1.in‚0 disk.sda3.out‚ 486.63671875K”¹V4’,/ _wandbruntime‚11998  _runtime‚11998TCtÈ“, _runtime‚11998.903562768 _step‚1159 % lr-AdamW/pg1‚9.80647620027288e-05 % lr-AdamW/pg2‚9.80647620027288e-05 trainer/global_step‚14099 % _timestamp‚1.7842848067779589e+09¢ýIâÍ”,Ç % lr-AdamW/pg1‚9.80647620027288e-05 % lr-AdamW/pg2‚9.80647620027288e-05 trainer/global_step‚14099 % _timestamp‚1.7842848067779589e+09 _runtime‚11998.903562768 _step‚1159‡ qD’M4•,/ _wandbruntime‚11998  _runtime‚11998~κO~–,P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24698581 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55063706 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚472026397616128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚472026379003904 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2158586917618688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14917384722432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111838372 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2158586917618688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130613594 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55063451 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2143669191557120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2158586559240192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15083117880832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487109496884736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130613594 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15083200166400 % _timestamp‚1.7842848067789736e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242451964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2158586559240192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2143669174517760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2147533582996826 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130623571 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111838370 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130623571 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30365125 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2147533940586076 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚14099 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2143669191557120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_Ùì$ ‰ batch_start/allocation.all.allocated‚ 242461943 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14894630331738 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14917384722432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚11998.904748707 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111838370 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111838372 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30364873 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2132638969704448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14917726061568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2132638952665088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2143669174517760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24698578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14894970881628 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2158586917618688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130613594 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55063451 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2143669191557120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2158586559240192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15083117880832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487109496884736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130613594 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15083200166400 % _timestamp‚1.7842848067789736e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242451964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2158586559240192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2143669174517760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2147533582996826 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130623571 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111838370 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130623571 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30365125 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2147533940586076 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚14099 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2143669191557120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242461943 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14894630331738 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14917384722432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚11998.904748707 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111838370 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111838372 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30364873 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2132638969704448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14917726061568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2132638952665088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2143669174517760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24698578 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14894970881628 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14918397349376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2158735171282432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130632637 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2143816432586752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130622646 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15084133271552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚472059807762432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111846002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14918738695680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242478641 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2132785480376320 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111846004 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14895641385460 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24700326 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55067250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242478641 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24700323 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111846004 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487144052410368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130632637 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30367193 _step‚1161 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2158734812896768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2147681462311782 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2147681104722420 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2158735171282432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30366927 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14895981935462 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14918397349376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242468648uÑþX#š,N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚472059826374656 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2132785463336960 % _timestamp‚1.7842848073904376e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242468648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55067250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242478641 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24700323 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111846004 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487144052410368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130632637 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30367193 _step‚1161 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2158734812896768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2147681462311782 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2147681104722420 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2158735171282432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30366927 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14895981935462 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14918397349376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242468648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55067519 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2143816415547392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14918397349376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2158735171282432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130632637 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2143816432586752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130622646 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15084133271552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚472059807762432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111846002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14918738695680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242478641 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2132785480376320 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111846004 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14895641385460 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peéâ ak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24700326 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14968421563392 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112219972 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473697835542528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14968080224256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30469204 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15134560366080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473697816930304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1164 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131068046 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2165999664602112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243297995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112219972 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131078023 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2165999306223616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚14149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 243288016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14968080224256 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24785828 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2151031225999360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2154909733077334 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2139964503298048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14968421563392 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112219972 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473697835542528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14968080224256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30469204 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15134560366080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473697816930304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1164 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131068046 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2165999664602112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243297995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112219972 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131078023 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2165999306223616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚14149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 243288016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14968080224256 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24785828 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2151031225999360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2154909733077334 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2139964503298048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15135505005568 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2155057254802928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 243314693 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2151178484068352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112227602 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55258865e«¢ 0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2140110996930560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243314693 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14969092851200 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2151178484068352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473731245688832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚14149 % _timestamp‚1.7842848472857983e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112227604 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131087089 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112227602 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0„Ë––'¬,N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2140110996930560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243314693 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14969092851200 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2151178484068352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473731245688832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚14149 % _timestamp‚1.7842848472857983e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112227604 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131087089 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112227602 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55259124 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473731264301056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15135587283968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14946598422370 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2155057612392290 _step‚1165 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14969092851200 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15135505005568 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2155057254802928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 243314693 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2151178484068352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112227602 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55258860 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 å]Ï4­,/ _wandbruntime‚12039  _runtime‚12039,Z;H®,‰ /loss/FBt1/detection_loss‚0.3300884543803242 /loss/FBt2/detection_loss‚0.3833950753273118 _step‚1166 3loss/SRqqt1/assignment_loss‚0.36369430607343484 3loss/FBt2/assignment_loss‚0.0011822976133318696 /loss/FRt2/detection_loss‚1.1929096427262493 1loss/SRqqt1/detection_loss‚0.8853498549476743 1loss/SRqqt2/detection_loss‚0.5798688391143897 _runtime‚12039.41408885 0loss/FRt1/assignment_loss‚1.5123048026098984 0loss/FRt2/assignment_loss‚0.7324079807149246 %loss/total_loss‚6.941205622101304 epoch‚35 % _timestamp‚1.7842848472887616e+09 1loss/FBt1/assignment_loss‚0.06053186207110385 trainer/global_step‚14149 3loss/SRqqt2/assignment_loss‚0.44786084954809907 0loss/FRt1/detection_loss‚0.45161165697456207“Øçþ¯,Ž 3loss/SRqqt2/assignment_loss‚0.44786084954809907 0loss/FRt1/detection_loss‚0.45161165697456207 /loss/FBt1/detection_loss‚0.3300884543803242 /loss/FBt2/detection_loss‚0.3833950753273118 _step‚1166 3loss/SRqqt1/assignment_loss‚0.36369430607343484 3loss/FBt2/assignment_loss‚0.0011822976133318696 /loss/FRt2/detection_loss‚1.1929096427262493 1loss/SRqqt1/detection_loss‚0.8853498549476743 1loss/SRqqt2/detection_loss‚0.5798688391143897 _runtime‚12039.41408885 0loss/FRt1/assignment_loss‚1.5123048026098984 0loss/FRt2/assignment_loss‚0.7324079807149246 %loss/total_loss‚6.941205622101304 epoch‚35 % _timestamp‚1.7842848472887616e+09 1loss/FBt1/assignment_loss‚0.06053186207110385 trainer/global_step‚14149Ž Ž¥F“°,‚(:ˆ ´èÒžÖþ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚256.992' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?N(&þ±,‚(:ó ´èÒ­Å¥ disk.sda.out‚ 487.828125 disk.dm-0.out‚53532.32421875 network.sent‚ 2.2476864e+07 disk.rbd0.out‚0 disk.sda3.in‚0% disk.nvme0c0n1.out‚53532.32421875 disk.sda2.in‚0 disk.dm-1.out‚ 487.828125! disk.rbd1.out‚114.26220703125 disk.rbd3.out‚0 disk.dm-0.in‚144875.61328125$ memory_percent‚97.37331867218018# disk.nvme0n1.out‚53532.32421875# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744) disk./.usagePercent‚36.82946148221235# disk.nvme0n1.in‚144875.61328125 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144875.61328125 disk.rbd3.in‚0( proc.memory.availableMB‚ 430.35546875 proc.cpu.threads‚17$ disk./.usageGB‚548.7855911254883 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625 disk.sda2.out‚0 disk.sda3.out‚ 487.828125 disk.dm-1.in‚0 cpu‚26.02871134598811 network.recv‚ 3.211099e+06 disk.sda1.in‚0 disk.sda1.out‚0fÒ“±“²,‚(:ˆ ÃèÒ®ú©¢ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LÿÒí³,‚(:â ÃèÒرƤ network.recv‚ 3.215767e+06 disk.dm-1.in‚0 disk.dm-0.in‚ 144876.328125) proc.memory.percent‚17.22586154937744! disk.nvme0n1.in‚ 144876.328125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚488.125 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.2539972e+07 disk.sda.in‚0 proc.cpu.threads‚17# disk.nvme0n1.out‚53541.42578125 disk.sda1.out‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 430.2890625 cpu‚26.0283267664691 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 144876.328125 disk.sda.out‚488.125$ memory_percent‚97.37372398376465# proc.memory.rssMB‚ 2822.28515625! disk.rbd1.out‚114.26220703125 disk.dm-0.out‚53541.42578125$ disk./.usageGB‚548.7855987548828% disk.nvme0c0n1.out‚53541.42578125 disk.dm-1.out‚488.125) disk./.usagePercent‚36.82946199422742”ª£ž´,‚(:“ ÒèÒîÚΣ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r÷y/ûµ,‚(:ð ÒèÒË—Ó¤ disk.sda1.out‚0( disk.nvme0c0n1.out‚53544.48486328125 disk.sda3.out‚ 488.34375 disk.dm-1.in‚0 disk.dm-0.in‚ 144876.328125 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0$ disk./.usageGB‚548.7853546142578 disk.rbd0.out‚0 disk.sda.out‚ 488.34375 disk.sda2.out‚0$ memory_percent‚97.37377166748047% proc.memory.availableMB‚ 430.28125 cpu‚26.02794066026844& disk.nvme0n1.out‚53544.48486328125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 144876.328125 disk.sda2.in‚0! disk.rbd1.out‚114.32470703125# proc.memory.rssMB‚ 2822.28515625 network.sent‚ 2.2544975e+07) disk./.usagePercent‚36.82944560974523! disk.nvme0n1.in‚ 144876.328125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.out‚ 488.34375) proc.memory.percent‚17.22586154937744# disk.dm-0.out‚53544.48486328125 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd3.in‚0 network.recv‚ 3.219177e+06ÆÉh4¶,/ _wandbruntime‚12078  _runtime‚12078°÷ŽÆ·,À trainer/global_step‚14199 $ _timestamp‚1.784284886394335e+09 _runtime‚12078.51992682 _step‚1167 % lr-AdamW/pg1‚9.50105489533175e-05 % lr-AdamW/pg2‚9.50105489533175e-05ä |)˸,Å % lr-AdamW/pg2‚9.50105489533175e-05 trainer/global_step‚14199 $ _timestamp‚1.784284886394335e+09 _runtime‚12078.51992682 _step‚1167 % lr-AdamW/pg1‚9.50105489533175e-05 !à”4¹,/ _wandbruntime‚12078  _runtime‚12078ŸiÞ¼(º,P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112601570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2147290019852288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842848863953552e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15018775726080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚490555035660288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475369254856704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55447707 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2173412053207040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15019117065216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112601572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚12078.524370915 _step‚1168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131532475 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30574349 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2162285883157842 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2158393294520320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112601570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚475369273468928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2162286240747092 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2147290036891648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2173412053207040 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2173412411585536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24873081 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490555149140992 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244134047 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55447427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14996203855444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244124068 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2158393277480960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2158393294520320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24873078 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112601572 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚01Ϊýi»,†P WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15019117065216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_t<3=š£%rain_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112601572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚12078.524370915 _step‚1168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131532475 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30574349 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2162285883157842 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2158393294520320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112601570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚475369273468928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2162286240747092 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2147290036891648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2173412053207040 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2173412411585536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24873081 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490555149140992 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244134047 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55447427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14996203855444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244124068 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2158393277480960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2158393294520320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24873078 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112601572 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112601570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2147290019852288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842848863953552e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15018775726080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚490555035660288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475369254856704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55447707 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2173412053207040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 tAZ(4¼,/ _wandbruntime‚12079  _runtime‚12079ñS3f'½,‹N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24874823 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15020129699328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15186892919296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15186795960832 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2158540518510592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55451181 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244150745 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2147436547563520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2158540535549952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15019788353024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131531550 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30576631 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475402702227456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2158540518510592 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14997214909278 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244140752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244150745 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131531550 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2173560306863616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30576358 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15020129699328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚475402683615232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24874826 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2162433404883436 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244140752AyË'¾,N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2158540518510592 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14997214909278 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244140752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244150745 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131531550 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2173560306863616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30576358 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15020129699328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚475402683615232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24874826 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2162433404883436 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244140752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24874823 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15020129699328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15186892919296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15186795960832 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2158540518510592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55451181 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244150745 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2147436547563520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2158540535549952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15019788353024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131531550 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30576631 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475402702227456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088‘ €Sh­4¿,/ _wandbruntime‚12079  _runtime‚12079=u»À,Œ /loss/FRt1/detection_loss‚0.5169816571775893 trainer/global_step‚14199 4loss/FBt2/assignment_loss‚4.1377582743595465e-05 0loss/SRqqt1/detection_loss‚0.895095954214203 1loss/SRqqt2/detection_loss‚0.6048144317547425 0loss/FBt1/detection_loss‚0.42720050241124563 0loss/FBt2/detection_loss‚0.47004264395593875 % _timestamp‚1.7842848870105536e+09 .loss/FRt1/assignment_loss‚1.49465662549051 3loss/SRqqt2/assignment_loss‚0.38724998068414346 &loss/total_loss‚7.0633040707766375 epoch‚35 _runtime‚12079.135877866 _step‚1170 0loss/FRt2/assignment_loss‚0.7361659507299292 4loss/FBt1/assignment_loss‚0.00038289630064757723 /loss/FRt2/detection_loss‚1.1913092499373557 2loss/SRqqt1/assignment_loss‚0.3393628005375889;œ·ãÁ,‘ 4loss/FBt2/assignment_loss‚4.1377582743595465e-05 0loss/SRqqt1/detection_loss‚0.895095954214203 1loss/SRqqt2/detection_loss‚0.6048144317547425 0loss/FBt1/detection_loss‚0.42720050241124563 0loss/FBt2/detection_loss‚0.47004264395593875 % _timestamp‚1.7842848870105536e+09 .loss/FRt1/assignment_loss‚1.49465662549051 3loss/SRqqt2/assignment_loss‚0.38724998068414346 &loss/total_loss‚7.0633040707766375 epoch‚35 _runtime‚12079.135877866 _step‚1170 0loss/FRt2/assignment_loss‚0.7361659507299292 4loss/FBt1/assignment_loss‚0.00038289630064757723 /loss/FRt2/detection_loss‚1.1913092499373557 2loss/SRqqt1/assignment_loss‚0.3393628005375889 /loss/FRt1/detection_loss‚0.5169816571775893 trainer/global_step‚14199’ KX´Ç’Â,‚(:‡ áèÒýº€¥ gpu.0.gpu‚13.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 à³ÔÃ,‚(:÷ áèÒ°š°§" disk.nvme0c0n1.in‚ 144877.15625 disk.sda2.in‚0$ disk./.usageGB‚548.7853813171387 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.829447401797964 disk.nvme0n1.in‚ 144877.15625 disk.sda.in‚0# proc.memory.rssMB‚ 2822.28515625! disk.rbd1.out‚114.32470703125 disk.sda.out‚ 488.6796875 disk.sda3.out‚ 488.6796875 disk.dm-1.in‚0 cpu‚26.02756428382281 disk.rbd1.in‚0.0625& disk.nvme0n1.out‚53553.35400390625$ memory_percent‚97.37217426300049( proc.memory.availableMB‚ 430.54296875 network.recv‚ 3.222957e+06 disk.rbd3.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 488.6796875 disk.dm-0.in‚ 144877.15625 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.22586154937744 network.sent‚ 2.2549963e+07( disk.nvme0c0n1.out‚53553.35400390625# disk.dm-0.ouÑa^y.t‚53553.35400390625 disk.drbd2165.out‚0¿dù.‘Ä,‚(:† ðèÒåÝï¤ gpu.0.gpu‚63.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ0°êÅ,‚(:û ðèÒ½Æ÷¦ disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.in‚144877.87109375' disk.nvme0c0n1.out‚53560.2568359375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda3.in‚0! disk.rbd1.out‚114.44970703125' proc.memory.availableMB‚ 430.8515625 disk.rbd0.out‚0 disk.dm-1.out‚ 488.9140625 cpu‚26.02714127192491# disk.nvme0n1.in‚144877.87109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 488.9140625$ memory_percent‚97.37029075622559 network.recv‚ 3.227323e+06* disk./.usagePercent‚36.829444841722626% disk.nvme0n1.out‚53560.2568359375 disk.sda2.out‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744" disk.dm-0.out‚53560.2568359375# disk./.usageGB‚548.785343170166 disk.sda2.in‚0 disk.sda3.out‚ 488.9140625 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.dm-0.in‚144877.87109375 network.sent‚ 2.2613166e+07Âè¿4Æ,/ _wandbruntime‚12118  _runtime‚12118ôª„˜ÊÇ,Ä _runtime‚12118.564982434 _step‚1171 & lr-AdamW/pg1‚9.349700895326078e-05 & lr-AdamW/pg2‚9.349700895326078e-05 trainer/global_step‚14249 % _timestamp‚1.7842849264393861e+098–Ð+ÏÈ,É % _timestamp‚1.7842849264393861e+09 _runtime‚12118.564982434 _step‚1171 & lr-AdamW/pg1‚9.349700895326078e-05 & lr-AdamW/pg2‚9.349700895326078e-05 trainer/global_step‚14249“ %»vC4É,/ _wandbruntime‚12118  _runtime‚12118×\d(Ê,€P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112983170 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112983172 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2180824800190464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492277654478848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15236943083520 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2180824800190464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30678193 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112983172 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15236854506496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2169662390827600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15046479792462 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2154615553445888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15069471227904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2180825158568960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112983170 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55638266 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131986927 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24960328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784284926440404e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131976950 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244970099 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131986927 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚12118.566434625 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15069812567040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2180825158568960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112983172 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2180824800190464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492277654478848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15236943083520 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2180824800190464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30678193 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112983172 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15236854506496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2169662390827600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15046479792462 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2154615553445888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15069471227904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2180825158568960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112983170 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55638266 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131986927 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24960328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784284926440404e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131976950 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244970099 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131986927 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚12118.566434625 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15069812567040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2180825158568960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244986797 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2165902569992192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15070483854848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15237883885056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492312110511616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55642034 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2165902569992192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2180973412232704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244986797 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112990804 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2180973412232704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24962073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24962076 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112990802 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131995993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244976804 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30680211 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚12119.181611416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131986002 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2165902587031552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30679961 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842849270555403e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112990804 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2180973053847040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15047831396186 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1173 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244986797 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112990804 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2180973412232704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24962073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24962076 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112990802 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131995993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244976804 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30680211 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚12119.181611416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131986002 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2165902587031552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30679961 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842849270555403e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112990804 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2180973053847040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15047831396186 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1173 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244986797 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2165902569992192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15070483854848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15237883885056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492312110511616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55642034 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2165902569992192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2180973412232704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0• „+¾z4Ï,/ _wandbruntime‚12119  _runtime‚12119‡BoYÐ,ˆ 3loss/SRqqt1/assignment_loss‚0.34786138847123466 /loss/FBt2/detection_loss‚0.5892420413230346 %loss/total_loss‚6.944700548854229 % _timestamp‚1.7842849270583248e+09 /loss/FBt1/detection_loss‚0.2951569739851125 epoch‚35 3loss/SRqqt2/assignment_loss‚0.45049667903222707 3loss/FBt1/assignment_loss‚0.0023644850374044866 1loss/FBt2/assignment_loss‚0.02808182861201252 0loss/FRt1/detection_loss‚0.48233363868054724 trainer/global_step‚14249 _runtime‚12119.183871752 1loss/SRqqt2/detection_loss‚0.6202750017521366 /loss/FRt1/assignment_loss‚1.449364886538423 0loss/FRt2/assignment_loss‚0.7543661135214347 /loss/FRt2/detection_loss‚1.1400619487161747 0loss/SRqqt1/detection_loss‚0.785095563184486 _step‚1174ºò,^Ñ, /loss/FBt2/detection_loss‚0.5892420413230346 %loss/total_loss‚6.944700548854229 % _timestamp‚1.7842849270583248e+09 /loss/FBt1/detection_loss‚0.2951569739851125 epoch‚35 3loss/SRqqt2/assignment_loss‚0.45049667903222707 3loss/FBt1/assignment_loss‚0.0023644850374044866 1loss/FBt2/assignment_loss‚0.02808182861201252 0loss/FRt1/detection_loss‚0.48233363868054724 trainer/global_step‚14249 _runtime‚12119.183871752 1loss/SRqqt2/detection_loss‚0.6202750017521366 /loss/FRt1/assignment_loss‚1.449364886538423 0loss/FRt2/assignment_loss‚0.7543661135214347 /loss/FRt2/detection_loss‚1.1400619487161747 0loss/SRqqt1/detection_loss‚0.785095563184486 _step‚1174 3loss/SRqqt1/assignment_loss‚0.34786138847123466– „#kÒ,‚(:’ ÿèÒ§åÅ¢ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚245.093' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚49.01859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·þ·0 Ó,‚(:‚ ÿèÒ˜…¦ disk.dm-1.out‚ 489.23046875) proc.memory.percent‚17.22586154937744! disk.rbd1.out‚114.44970703125# disk.dm-0.out‚53571.02978515625 proc.cpu.threads‚17( disk./.usagePercent‚36.8294558500466 disk.sda3.out‚ 489.23046875# proc.memory.rssMB‚ 2822.28515625 disk.sda3.in‚0 network.recv‚ 3.230951e+06 disk.sda1.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚53571.02978515625 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144878.69921875 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚144878.69921875$ memory_percent‚97.37398624420166( proc.memory.availableMB‚ 430.24609375 cpu‚26.026762511297328# disk.nvme0n1.in‚144878.69921875 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚53571.02978515625 network.sent‚ 2.2618273e+07$ disk./.usageGB‚548.7855072021484 disk.rbd0.out‚0 disk.sda.out‚ 489.23046875 disk.dm-1.in‚0*œ“Ô,‚(:ˆ ŽŽèÒùœ¸¢ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚250.216' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.0432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L‘ eÕ,‚(:ý ŽŽèÒµÈÛ¤" disk.nvme0n1.in‚144879.4140625 disk.dm-1.out‚ 489.54296875 cpu‚26.026386497280427) disk./.usagePercent‚36.82945405799386 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚53577.7802734375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.22586154937744 network.sent‚ 2.2681333e+07 disk.sda1.out‚0" disk.dm-0.out‚53577.7802734375' disk.nvme0c0n1.out‚53577.7802734375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 489.54296875 disk.rbd1.in‚0.0625 disk.sda3.out‚ 489.54296875! disk.rbd1.out‚114.51220703125( proc.memory.availableMB‚ 430.20703125 network.recv‚ 3.235125e+06 disk.sda.in‚0 disk.dm-0.in‚144879.4140625 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚144879.4140625 disk.rbd3.in‚0$ memory_percent‚97.37422466278076# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17$ disk./.usageGB‚548.7854804992676×·Ìý“Ö,‚(:ˆ ŽèÒÑù–¢ gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚278.522' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04bc¢þ×,‚(:ó ŽèÒÅäÃ¥ proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625 network.recv‚ 3.238559e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚53583.9169921875" disk.nvme0n1.in‚144879.4140625$ memory_percent‚97.37427234649658# proc.memory.rssMB‚ 2822.28515625 disk.rbd3.in‚0 disk.dm-0.in‚144879.4140625 network.sent‚ 2.2686407e+07) disk./.usagePercent‚36.82946097019728' disk.nvme0c0n1.out‚53583.9169921875 disk.sda3.out‚ 489.71875 disk.dm-1.out‚ 489.71875) proc.memory.percent‚17.22586154937744 disk.rbd3.out‚0$ disk./.usageGB‚548.7855834960938 disk.drbd2165.out‚0 disk.sda.out‚ 489.71875" disk.dm-0.out‚53583.9169921875( proc.memory.availableMB‚ 430.19921875 cpu‚26.02602245174179$ disk.nvme0c0n1.in‚144879.4140625 disk.sda2.in‚0 disk.sda3.in‚0! disk.rbd1.out‚114.51220703125Iq¤4Ø,/ _wandbruntime‚12158  _runtime‚12158Ÿ†€©ÊÙ,Ä & lr-AdamW/pg2‚9.199264311309621e-05 trainer/global_step‚14299 % _timestamp‚1.7842849664303913e+09 _runtime‚12158.557376134 _step‚1175 & lr-AdamW/pg1‚9.199264311309621e-05¢ãÏÚ,É _runtime‚12158.557376134 _step‚1175 & lr-AdamW/pg1‚9.199264311309621e-05 & lr-AdamW/pg2‚9.199264311309621e-05 trainer/global_step‚14299 % _timestamp‚1.7842849664303913e+09— ½ C!4Û,/ _wandbruntime‚12158  _runtime‚12158(‚°2(Ü,€P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚14299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15120508068864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15097436829260 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132441379 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2188237547173888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132431402 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2173117397483520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚12158.557457317 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132441379 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15288204271104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55829247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚494000248500736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2161941087039488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2188237547173888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25047581 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2161941104078848 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25047578 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478712130709504˜M¤ØÝ,…P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2188237547173888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25047581 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2161941104078848 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25047578 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478712130709504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚14299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15120508068864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15097436829260 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132441379 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2188237547173888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132431402 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2173117397483520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚12158.557457317 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132441379 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15288204271104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55829247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚494000248500736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2161941087039488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0˜ è¹4Þ,/ _wandbruntime‚12159  _runtime‚12159O"Sé'ß,‹N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113372402 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2188386159216128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15289142090240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2177186062633814 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15121520702976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚14299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 245822849 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2177185705044452 _runtime‚12159.167911324 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15121179356672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15289226465792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2188385800830464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15098107333092 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478745578080256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245812856 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15121179356672 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113372404 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55832780 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245822849 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚1177 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30783715 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113372404 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2162087614750720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15121520702976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2173264638513152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 245812856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600$s3ƒ'à,N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245822849 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚1177 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30783715 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113372404 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2162087614750720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15121520702976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2173264638513152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 245812856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113372402 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2188386159216128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15289142090240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2177186062633814 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15121520702976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚14299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 245822849 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2177185705044452 _runtime‚12159.167911324 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15121179356672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15289226465792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2188385800830464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15098107333092 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478745578080256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245812856 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15121179356672 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113372404 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55832780 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15171203570688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113746372 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2180479448965120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7842850062248197e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56019708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30884877 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15171203570688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2195650294157312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246642203 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15339220352000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2180479448965120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56019439 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚12198.350597146 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2195650294157312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132895831 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495722788987904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚480383587248128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15339306831872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113746370 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚14349 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132895831 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2180479431925760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15171203570688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113746372 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2180479448965120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7842850062248197e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56019708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30884877 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15171203570688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2195650294157312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246642203 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15339220352000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2180479448965120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56019439 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚12198.350597146 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2195650294157312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132895831 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495722788987904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚480383587248128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15339306831872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113746370 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚14349 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132895831 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2180479431925760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15340247723520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2195798906199552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2180626689994752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113754004 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2180626672955392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246658901 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56023465 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132894906 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15171874858496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246658901 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚12198.960953929 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15171874858496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15172216204800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246648908 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2169413148344320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚480417016006656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2184561855124960 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2180626689994752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2195798906199552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56023206 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132904897 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2184562212714322 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30886889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842850068351648e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15340332099072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25136576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495757245117952­†[ò,N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2180626689994752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2195798906199552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56023206 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132904897 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2184562212714322 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30886889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842850068351648e+09 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15340332099072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25136576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495757245117952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15340247723520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2195798906199552 _JDeviceStatsMonitor.on_train_batcÍ™«A»h_end/allocated_bytes.large_pool.allocated‚2180626689994752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113754004 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1181 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2180626672955392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246658901 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56023465 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132894906 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15171874858496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246658901 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚12198.960953929 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15171874858496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15172216204800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246648908 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2169413148344320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚480417016006656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2184561855124960 šw¾¤4ó,/ _wandbruntime‚12198  _runtime‚12198@ˆ­—ô,ˆ 0loss/FRt1/detection_loss‚0.47405565152247353 1loss/SRqqt1/detection_loss‚0.8405297959184241 /loss/FBt1/detection_loss‚0.4157969077994214 0loss/FRt1/assignment_loss‚1.4822824450450405 0loss/FRt2/assignment_loss‚0.7380595240575164 .loss/FRt2/detection_loss‚1.179121901516615 _runtime‚12198.964092061 2loss/SRqqt2/assignment_loss‚0.4816646261672353 3loss/FBt1/assignment_loss‚0.0031882536957638166 /loss/FBt2/detection_loss‚0.5974772908278586 _step‚1182 2loss/FBt2/assignment_loss‚0.014819416348824296 0loss/SRqqt2/detection_loss‚0.573718585677232 %loss/total_loss‚7.175812834498808 epoch‚35 trainer/global_step‚14349 % _timestamp‚1.7842850068387547e+09 3loss/SRqqt1/assignment_loss‚0.37509843592240427Pªnæõ, 2loss/FBt2/assignment_loss‚0.014819416348824296 0loss/SRqqt2/detection_loss‚0.573718585677232 %loss/total_loss‚7.175812834498808 epoch‚35 trainer/global_step‚14349 % _timestamp‚1.7842850068387547e+09 3loss/SRqqt1/assignment_loss‚0.37509843592240427 0loss/FRt1/detection_loss‚0.47405565152247353 1loss/SRqqt1/detection_loss‚0.8405297959184241 /loss/FBt1/detection_loss‚0.4157969077994214 0loss/FRt1/assignment_loss‚1.4822824450450405 0loss/FRt2/assignment_loss‚0.7380595240575164 .loss/FRt2/detection_loss‚1.179121901516615 _runtime‚12198.964092061 2loss/SRqqt2/assignment_loss‚0.4816646261672353 3loss/FBt1/assignment_loss‚0.0031882536957638166 /loss/FBt2/detection_loss‚0.5974772908278586 _step‚1182ž Fñl¥“ö,‚(:ˆ ÙŽèÒ±º½¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚248.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0md1p ÷,‚(: ÙŽèÒÙÇ㤠disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚53611.93505859375 network.recv‚ 3.253134e+06 disk.dm-0.in‚144881.78515625% disk.nvme0c0n1.in‚144881.78515625 disk.sda.in‚0 disk.dm-1.out‚ 490.7734375 disk.rbd1.in‚0.0625 proc.cpu.threads‚17 disk.rbd3.in‚0$ memory_percent‚97.37603664398193) proc.memory.percent‚17.22586154937744 network.sent‚ 2.2764537e+07 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 429.91015625# proc.memory.rssMB‚ 2822.28515625 cpu‚26.024490830000634* disk./.usagePercent‚36.829443049669884$ disk./.usageGB‚548.7853164672852 disk.sda2.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚144881.78515625 disk.rbd0.out‚0 disk.sda.out‚ 490.7734375 disk.sda2.out‚0 disk.sda3.in‚0! disk.rbd1.out‚114.63720703125# disk.dm-0.out‚53611.93505859375& disk.nvme0n1.out‚53611.93505859375 disk.sda3.out‚ 490.7734375 disk.dm-1.in‚0meåeœø,‚(:‘ èŽèÒÏᢠgpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.333600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šh>–åù,‚(:Ú èŽèÒ£…–¥$ disk./.usageGB‚548.7851982116699! disk.nvme0n1.out‚ 53619.859375 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 53619.859375 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚144882.5) proc.memory.percent‚17.22586154937744 disk.dm-0.out‚ 53619.859375 disk.sda.out‚ 491.109375! disk.rbd1.out‚114.69970703125 cpu‚26.024112790463395* disk./.usagePercent‚36.829435113436325 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 430.60546875# proc.memory.rssMB‚ 2822.28515625 network.recv‚ 3.257429e+06 disk.dm-0.in‚144882.5 disk.sda3.out‚ 491.109375 disk.dm-1.out‚ 491.109375 disk.sda1.in‚0 disk.nvme0c0n1.in‚144882.5 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚97.37179279327393 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 network.sent‚ 2.2829805e+07º‡4ú,/ _wandbruntime‚12238  _runtime‚12238Þ,JÅÊû,Ä trainer/global_step‚14399 % _timestamp‚1.7842850461856916e+09 _runtime‚12238.311392958 _step‚1183 & lr-AdamW/pg1‚8.901181980742642e-05 & lr-AdamW/pg2‚8.901181980742642e-05û@ÅãÏü,É _runtime‚12238.311392958 _step‚1183 & lr-AdamW/pg1‚8.901181980742642e-05 & lr-AdamW/pg2‚8.901181980742642e-05 trainer/global_step‚14399 % _timestamp‚1.7842850461856916e+09Ÿ €W¶‹4ý,/ _wandbruntime‚12238  _runtime‚12238Oñ(þ,ÿO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2203063399519232 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2203063041140736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2203063399519232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15221557733376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2187841483407360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247468276 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30988782 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15390303875584 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30988518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2176592171266048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56210596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2203063041140736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497445310437888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2176592154226688 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247478255 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2191790841069124 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25222078 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114127972 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25222081 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15198329253186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114127970 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15221899072512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚12238.316350089 # _timestamp‚1.78428504618683e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15198669803076 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚1184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497445419724288 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2191790483479874 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2187841500446720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133340306 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56210863 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088xø.n (ÿ,„P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497445310437888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2176592154226688 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247478255 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2191790841069124 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25222078 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114127972 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25222081 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15198329253186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114127970 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15221899072512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚12238.316350089 # _timestamp‚1.78428504618683e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15198669803076 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚1184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497445419724288 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2191790483479874 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2187841500446720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133340306 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56210863 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2203063399519232 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2203063041140736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2203063399519232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15221557733376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2187841483407360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247468276 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30988782 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15390303875584 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30988518 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2176592171266048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56210596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2203063041140736  ù.Õ4€-/ _wandbruntime‚12238  _runtime‚12238Ø|£9-‹N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482088453933056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497479869109248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.sm’¼\Võ!all_pool.allocated‚15222911706624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114135602 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56214393 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247484960 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15222911706624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15391415176192 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114135604 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7842850467996397e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15222570360320 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2187988741476352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚14399 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114135604 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2191938005205468 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2176738681937920 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2176738664898560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15199340306908 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2187988724436992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15391324509184 _runtime‚12238.925452821 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25223826 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25223823 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133349358 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2203211653182976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30990570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15199680856910 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2187988741476352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247494953 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56214667 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133359349 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247484960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482088435320832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15222570360320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133349358 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133359349 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247484960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482088435320832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15222570360320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133349358 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482088453933056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497479869109248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15222911706624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114135602 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56214393 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247484960 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15222911706624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15391415176192 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114135604 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7842850467996397e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15222570360320 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2187988741476352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚14399 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114135604 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2191938005205468 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2176738681937920 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2176738664898560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15199340306908 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2187988724436992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15391324509184 _runtime‚12238.925452821 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25223826 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25223823 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133349358 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2203211653182976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30990570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15199680856910 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2187988741476352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247494953 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¡ äëP4ƒ-/ _wandbruntime‚12238  _runtime‚12238#9²Þ„-ˆ /loss/FRt1/detection_loss‚0.4925815983585477 %loss/total_loss‚6.811128649068468 epoch‚35 3loss/SRqqt1/assignment_loss‚0.34035551196020786 3loss/SRqqt2/assignment_loss‚0.39443321877689563 /loss/FRt2/detection_loss‚1.1909276446247887 1loss/SRqqt2/detection_loss‚0.6032153179409326 /loss/FBt1/detection_loss‚0.2682551925862126 trainer/global_step‚14399 .loss/FRt1/assignment_loss‚1.44122953308868 0loss/FRt2/assignment_loss‚0.7810747841229806 2loss/FBt1/assignment_loss‚0.005497086911532879 /loss/FBt2/detection_loss‚0.3688173921645752 _runtime‚12238.927658903 _step‚1186 3loss/FBt2/assignment_loss‚0.0023951508604493402 1loss/SRqqt1/detection_loss‚0.9223462176726647 % _timestamp‚1.7842850468023252e+09ùõ…- 1loss/SRqqt1/detection_loss‚0.9223462176726647 % _timestamp‚1.7842850468023252e+09 /loss/FRt1/detection_loss‚0.4925815983585477 %loss/total_loss‚6.811128649068468 epoch‚35 3loss/SRqqt1/assignment_loss‚0.34035551196020786 3loss/SRqqt2/assignment_loss‚0.39443321877689563 /loss/FRt2/detection_loss‚1.1909276446247887 1loss/SRqqt2/detection_loss‚0.6032153179409326 /loss/FBt1/detection_loss‚0.2682551925862126 trainer/global_step‚14399 .loss/FRt1/assignment_loss‚1.44122953308868 0loss/FRt2/assignment_loss‚0.7810747841229806 2loss/FBt1/assignment_loss‚0.005497086911532879 /loss/FBt2/detection_loss‚0.3688173921645752 _runtime‚12238.927658903 _step‚1186 3loss/FBt2/assignment_loss‚0.0023951508604493402¢ m¨›‘†-‚(:† ÷ŽèÒÞ¯ò¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î|î ‡-‚(:þ ÷ŽèÒ¸íÀ¤ disk.rbd3.in‚0# disk.dm-0.out‚53630.44482421875 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚53630.44482421875 disk.rbd1.in‚0.0625 cpu‚26.024025249356995 disk.sda.out‚ 491.47265625! disk.nvme0n1.in‚ 144883.328125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2822.28515625 disk.rbd3.out‚0 network.recv‚ 3.260921e+06 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 144883.328125 disk.sda3.out‚ 491.47265625 disk.dm-1.out‚ 491.47265625! disk.rbd1.out‚114.69970703125( proc.memory.availableMB‚ 430.83984375$ disk./.usageGB‚548.7852554321289 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚97.37036228179932) proc.memory.percent‚17.22586154937744 proc.cpu.threads‚17 network.sent‚ 2.2832769e+07* disk./.usagePercent‚36.829438953549335& disk.nvme0n1.out‚53630.44482421875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 144883.328125 disk.drbd2165.out‚0[¡%žˆ-‚(:“ †èÒ륑¤ gpu.0.gpu‚72.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.007600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0> ׉-‚(:ý †èÒÈ disk.drbd2165.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚548.7850379943848 disk.sda3.out‚ 491.7109375( proc.memory.availableMB‚ 430.83203125# proc.memory.rssMB‚ 2822.28515625 network.recv‚ 3.26532e+06 cpu‚26.028302180781647% disk.nvme0c0n1.in‚144884.04296875 disk.rbd3.out‚0$ memory_percent‚97.37040996551514 network.sent‚ 2.2895807e+07* disk./.usagePercent‚36.829424361119884 disk.rbd0.in‚0' disk.nvme0c0n1.out‚53637.2080078125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚53637.2080078125 disk.sda3.in‚0 disk.dm-1.out‚ 491.7109375 disk.rbd1.in‚0.0625" disk.dm-0.out‚53637.2080078125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 491.7109375! disk.rbd1.out‚114.82470703125 disk.dm-0.in‚144884.04296875) proc.memory.percent‚17.22586154937744# disk.nvme0n1.in‚144884.04296875 disk.sda1.out‚0B”«4Š-/ _wandbruntime‚12255  _runtime‚12255ä{Ûß$‹-š{ =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03762102872133255 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.333977073431015 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6249786019325256 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.206330806016922 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6674164533615112 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 _step‚1187 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6041983366012573 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6870369911193848 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02364223822951317 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30621638894081116 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7312855124473572 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04417571425437927 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7236838340759277 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983811378479004 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02257935144007206 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5337550640106201 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518516421318054 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8992063403129578 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6316544413566589 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6906145811080933 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7620936632156372 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6609116792678833 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.770370364189148 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6320462226867676 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27811896800994873 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01937750168144703 particle/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7221155762672424 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6189925670623779 particle/accuracy_1_of_2‚1 )loss/val_total_loss‚7.907215728619947 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3396443724632263 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5291666388511658 *jet/accuracy_2_of_4‚0.7972971796989441 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36134371161460876 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8451087474822998 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996785581111908 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21809811890125275 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_2_of_3‚0.5710026621818542 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6327546238899231 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6610378623008728 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21398179233074188 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6643061637878418 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5495396852493286 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 _runtime‚12255.972339386 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995650589466095 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2785016596317291 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25992846488952637 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2886856198310852 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2621937394142151 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03601239621639252 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2596161365509033 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9869480133056641 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07098939269781113 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5778185725212097 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6029211282730103 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5163630843162537 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960803985595703 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9930327534675598 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5027247667312622 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5991920232772827 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6092936396598816 particle/accuracy_2_of_2‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879798889160156 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2785016596317291 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7551209330558777 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9979296326637268 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22690856456756592 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4550001919269562 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148059844971 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6271737217903137 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6628211140632629 *jet/accuracy_1_of_3‚0.9261393547058105 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997230052947998 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37311848998069763 'particle/f_score‚0.6245431303977966 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8617099523544312 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.633082926273346 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.860887348651886 *jet/accuracy_1_of_1‚0.3368648886680603 +particle/sensitivity‚0.6128524541854858 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2888343036174774 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6293148994445801 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48228344321250916 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01937750168144703 particle/accuracy_1_of_4‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.927219569683075 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28825002908706665 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.860887348651886 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22783760726451874 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.08632738143205643 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9226319193840027 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 particle/accuracy_2_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909415245056152 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6109331846237183 *jet/accuracy_1_of_4‚0.9768441319465637 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚14399 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt1F6\¤CÁBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024794243276119232 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875695109367371 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6109331846237183 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2205825299024582 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9852258563041687 +jet/accuracy_2_of_2‚0.15826335549354553 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9534449577331543 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36136749386787415 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946331977844238 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 'particle/accuracy‚0.841773509979248 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6405367851257324 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08856888115406036 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023075774312019348 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7551209330558777 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5667246580123901 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.650614857673645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5783802270889282 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2403750866651535 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2251865565776825 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7939776182174683 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7608850002288818 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +particle/specificity‚0.9043773412704468 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 % _timestamp‚1.7842850638460348e+09 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9926129579544067 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚35 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7601862549781799 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_1_of_1‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7921567559242249 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7705971598625183 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7731000781059265 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6939485669136047 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24127638339996338 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24892565608024597 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7884032130241394 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.270740270614624 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975825548171997 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2515856623649597 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2708902060985565 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2885534167289734 +jet/accuracy_3_of_3‚0.06053808331489563 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24499034881591797 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20976191759109497 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17083333432674408 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6609731912612915 jet/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26963573694229126 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.599202573299408 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7662556171417236 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04662758484482765 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6210635900497437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6validation_average_jet_accuracy‚0.3784690499305725Q°dÖ¥=Œ-Ÿ{ & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7884032130241394 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.270740270614624 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975825548171997 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2515856623649597 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2708902060985565 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2885534167289734 +jet/accuracy_3_of_3‚0.06053808331489563 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24499034881591797 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20976191759109497 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17083333432674408 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6609731912612915 jet/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26963573694229126 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.599202573299408 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7662556171417236 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04662758484482765 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6210635900497437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6validation_average_jet_accuracy‚0.3784690499305725 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03762102872133255 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.333977073431015 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6249786019325256 6 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.206330806016922 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6674164533615112 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 _step‚1187 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6041983366012573 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6870369911193848 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02364223822951317 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30621638894081116 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7312855124473572 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04417571425437927 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7236838340759277 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983811378479004 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02257935144007206 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5337550640106201 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518516421318054 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8992063403129578 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6316544413566589 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6906145811080933 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7620936632156372 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6609116792678833 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.770370364189148 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6320462226867676 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27811896800994873 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01937750168144703 particle/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7221155762672424 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6189925670623779 particle/accuracy_1_of_2‚1 )loss/val_total_loss‚7.907215728619947 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3396443724632263 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5291666388511658 *jet/accuracy_2_of_4‚0.7972971796989441 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36134371161460876 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8451087474822998 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996785581111908 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21809811890125275 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_2_of_3‚0.5710026621818542 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6327546238899231 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6610378623008728 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21398179233074188 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6643061637878418 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5495396852493286 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 _runtime‚12255.972339386 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995650589466095 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2785016596317291 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25992846488952637 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2886856198310852 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2621937394142151 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03601239621639252 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2596161365509033 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9869480133056641 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07098939269781113 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5778185725212097 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6029211282730103 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5163630843162537 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960803985595703 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9930327534675598 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5027247667312622 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5991920232772827 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6092936396598816 particle/accuracy_2_of_2‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879798889160156 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2785016596317291 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7551209330558777 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9979296326637268 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22690856456756592 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4550001919269562 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148059844971 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6271737217903137 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6628211140632629 *jet/accuracy_1_of_3‚0.9261393547058105 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997230052947998 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37311848998069763 'particle/f_score‚0.6245431303977966 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8617099523544312 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.633082926273346 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.860887348651886 *jet/accuracy_1_of_1‚0.3368648886680603 +particle/sensitivity‚0.6128524541854858 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2888343036174774 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6293148994445801 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48228344321250916 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01937750168144703 particle/accuracy_1_of_4‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.927219569683075 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28825002908706665 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.860887348651886 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22783760726451874 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.08632738143205643 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9226319193840027 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 particle/accuracy_2_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909415245056152 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6109331846237183 *jet/accuracy_1_of_4‚0.9768441319465637 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚14399 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024794243276119232 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875695109367371 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6109331846237183 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2205825299024582 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9852258563041687 +jet/accuracy_2_of_2‚0.15826335549354553 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9534449577331543 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36136749386787415 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946331977844238 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 'particle/accuracy‚0.841773509979248 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6405367851257324 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08856888115406036 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023075774312019348 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7551209330558777 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5667246580123901 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.650614857673645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5783802270889282 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2403750866651535 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2251865565776825 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7939776182174683 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7608850002288818 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +particle/specificity‚0.9043773412704468 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 % _timestamp‚1.7842850638460348e+09 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9926129579544067 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚35 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7601862549781799 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_1_of_1‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7921567559242249 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7705971598625183 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7731000781059265 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6939485669136047 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24127638339996338 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24892565608024597£ ÛÑs“-‚(:ˆ •èÒÁÃŒ£ gpu.0.gpu‚53.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i~rè Ž-‚(: •èÒñþ¤& proc.memory.availableMB‚ 430.734375) proc.memory.percent‚17.22586154937744# disk.nvme0n1.in‚144884.87109375 network.recv‚ 3.268707e+06 disk.rbd0.in‚0 disk.sda3.out‚ 492.14453125 disk.dm-1.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚114.82470703125 disk.dm-0.in‚144884.87109375 disk.drbd2165.in‚0 disk.sda.out‚ 492.14453125 disk.sda3.in‚0 disk.dm-1.out‚ 492.14453125 disk.sda1.in‚0( disk.nvme0c0n1.out‚53646.13037109375 disk.sda2.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚53646.13037109375 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚26.028511716713933) disk./.usagePercent‚36.82942384910482$ disk./.usageGB‚548.7850303649902 disk.rbd0.out‚0# proc.memory.rssMB‚ 2822.28515625& disk.nvme0n1.out‚53646.13037109375 disk.drbd2165.out‚0$ memory_percent‚97.37100601196289 proc.cpu.threads‚17 network.sent‚ 2.2900921e+07 disk.rbd3.in‚0% disk.nvme0c0n1.in‚144884.87109375 disk.sda.in‚0tÙ_À“-‚(:ˆ ¤èÒ“‰Û¢ gpu.0.gpu‚52.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“--‚(:ø ¤èÒéú¤ disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚144884.87109375 disk.dm-0.out‚53649.54296875# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744# disk.nvme0n1.in‚144884.87109375% disk.nvme0c0n1.out‚53649.54296875 disk.dm-1.out‚ 492.3203125$ memory_percent‚97.37217426300049 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144884.87109375* disk./.usagePercent‚36.829471978521255$ disk./.usageGB‚548.7857475280762 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 3.273061e+06# disk.nvme0n1.out‚53649.54296875 disk.rbd0.in‚0 disk.sda.out‚ 492.3203125( proc.memory.availableMB‚ 430.54296875 network.sent‚ 2.2958568e+07 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 492.3203125! disk.rbd1.out‚114.88720703125 cpu‚26.028152735884433œþx±4‘-/ _wandbruntime‚12295  _runtime‚12295t`ïìÈ’- % lr-AdamW/pg1‚8.75355537946317e-05 % lr-AdamW/pg2‚8.75355537946317e-05 trainer/global_step‚14449 % _timestamp‚1.7842851031321945e+09 _runtime‚12295.257800128 _step‚1188—Qê*Í“-Ç trainer/global_step‚14449 % _timestamp‚1.7842851031321945e+09 _runtime‚12295.257800128 _step‚1188 % lr-AdamW/pg1‚8.75355537946317e-05 % lr-AdamW/pg2‚8.75355537946317e-05¤ ÁÏ_å4”-/ _wandbruntime‚12295  _runtime‚12295&c•-€P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2212423087980032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚483742356588544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2201093016499691 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚14449 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15290327779840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114630236 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2201093374088941 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133875057 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15290327779840 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31122709 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133865080 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.all/Ü2që ocated‚2197133118578688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚499201988039680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483742375200768 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2197133118578688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 $ _timestamp‚1.784285103133264e+09 _step‚1189 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25311851 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2185826368073728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248505295 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15267006015213 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15459526359040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚499201882947584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114630238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269ìU?õ (–-…P _step‚1189 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25311851 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2185826368073728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248505295 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15267006015213 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15459526359040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚499201882947584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114630238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2212423087980032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚483742356588544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2201093016499691 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚14449 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15290327779840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114630236 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2201093374088941 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133875057 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15290327779840 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31122709 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133865080 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2197133118578688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚499201988039680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483742375200768 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2197133118578688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 $ _timestamp‚1.784285103133264e+09¥ 5Ým`4—-/ _wandbruntime‚12295  _runtime‚12295œI»_'˜-‹N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248512000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133874132 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483775803959296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚12295.872322743 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2201240895814647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚14449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248521993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15291340413952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25313596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25313593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499236340985856 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15291340413952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499236446070784 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31124521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15290999067648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2197280342568960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133884123 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114637870 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2212571341636608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15290999067648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15460642111488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248521993 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15268017069047 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56438378 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2197280359608320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114637868 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2212571700022272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15460555638784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2197280342568960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842851037464275e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114637870 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31124782 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56438114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248521993 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15268017069047 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56438378 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2197280359608320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114637868 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2212571700022272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15460555638784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2197280342568960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842851037464275e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114637870 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31124782 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56438114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2201240895814647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚14449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248521993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15291340413952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25313596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25313593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499236340985856 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15291340413952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499236446070784 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31124521 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15290999067648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2197280342568960 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133884123 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114637870 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2212571341636608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚1861222Io×t4 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15290999067648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976¦ µoÓ4š-/ _wandbruntime‚12295  _runtime‚12295)qó›-‰ 1loss/SRqqt1/detection_loss‚0.7324196993639427 /loss/FBt1/detection_loss‚0.4542093545793594 0loss/FRt1/assignment_loss‚1.4432065238341176 3loss/SRqqt1/assignment_loss‚0.36136613711493093 epoch‚36 4loss/FBt2/assignment_loss‚0.00034778310359016016 /loss/FBt2/detection_loss‚0.6684861310602476 % _timestamp‚1.7842851037480476e+09 _step‚1191 2loss/SRqqt2/assignment_loss‚0.4316190544568693 /loss/FRt1/detection_loss‚0.5290997931915823 /loss/FRt2/detection_loss‚1.3655340693686235 1loss/SRqqt2/detection_loss‚0.6471123429800524 %loss/total_loss‚7.344028212002895 trainer/global_step‚14449 _runtime‚12295.87339059 0loss/FRt2/assignment_loss‚0.6972944457779231 2loss/FBt1/assignment_loss‚0.013332877171656184÷µ'#œ-Ž 2loss/SRqqt2/assignment_loss‚0.4316190544568693 /loss/FRt1/detection_loss‚0.5290997931915823 /loss/FRt2/detection_loss‚1.3655340693686235 1loss/SRqqt2/detection_loss‚0.6471123429800524 %loss/total_loss‚7.344028212002895 trainer/global_step‚14449 _runtime‚12295.87339059 0loss/FRt2/assignment_loss‚0.6972944457779231 2loss/FBt1/assignment_loss‚0.013332877171656184 1loss/SRqqt1/detection_loss‚0.7324196993639427 /loss/FBt1/detection_loss‚0.4542093545793594 0loss/FRt1/assignment_loss‚1.4432065238341176 3loss/SRqqt1/assignment_loss‚0.36136613711493093 epoch‚36 4loss/FBt2/assignment_loss‚0.00034778310359016016 /loss/FBt2/detection_loss‚0.6684861310602476 % _timestamp‚1.7842851037480476e+09 _step‚1191§ ,DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2208469524169449 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2219835834963456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 % _timestamp‚1.7842851428838232e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31226586 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115011838 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500924673716736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2208469166580199 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115011838 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485413813127168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚14499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15341023281664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249331368 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2204495170060288 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500924558138880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25399101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15341023281664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56625415 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249341347 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134329509 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2193151884627968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15340681942528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2204495170060288 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15317622502121 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15317281952231 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2204495153020928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15510763623936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134319532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56625687 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485413794514944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚12335.009609681 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134329509 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2219835834963456¦D@` (¨-†P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25399101 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15341023281664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56625415 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249341347 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134329509 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2193151884627968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15340681942528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2204495170060288 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15317622502121 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15317281952231 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2204495153020928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15510763623936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134319532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56625687 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485413794514944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚12335.009609681 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134329509 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2219835834963456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134319532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31226317 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15510860589568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2219836193341952 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2193151901667328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25399098 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2208469524169449 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2219835834963456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 % _timestamp‚1.7842851428838232e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31226586 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115011838 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500924673716736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2208469166580199 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115011838 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485413813127168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚14499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15341023281664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249331368 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2204495170060288 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500924558138880© ¥økŸ4©-/ _wandbruntime‚12335  _runtime‚12335Íœ%Ÿª-ŠN E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15511779690496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134328584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249348052 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2219984088620032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115019470 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249358045 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25400843 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12335.62202696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56629507 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500959002963968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2204642411089920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25400846 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2193298395299840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2219984447005696 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56629243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15318633555955 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15341694569472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31228661 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚14499 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2204642411089920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2208617045895155 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304å#J»q ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15342035915776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249348052 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500959114340352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15342035915776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115019468 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134338575 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249358045 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134328584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249348052 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2219984088620032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115019470 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249358045 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25400843 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12335.62202696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56629507 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500959002963968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2204642411089920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25400846 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2193298395299840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2219984447005696 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56629243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15318633555955 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15341694569472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31228661 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚14499 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2204642411089920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2208617045895155 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15342035915776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249348052 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500959114340352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15342035915776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115019468 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134338575 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249358045 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502647187215360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚14549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250177399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25486351 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2211857221541888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2211857221541888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250167420 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2227248581946880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115393436 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2227248581946880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56816853 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2215845316660707 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2227248940325376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15391377444352 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚502647077928960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚487085232441344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚487085251053568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2215845674249957 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115393436 _step‚1197 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15391718783488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2200477418221568 $ _timestamp‚1.784285182803735e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15368238989029 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2227248940325376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134773984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31330502 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15367898439139 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15391377444352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15391718783488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2200477435260928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115393438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176/^b‹v ¸-…P NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2200477435260928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115393438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502647187215360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 trainer/global_step‚14549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250177399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25486351 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2211857221541888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2211857221541888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250167420 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2227248581946880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115393436 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2227248581946880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56816853 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2215845316660707 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2227248940325376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15391377444352 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚502647077928960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚487085232441344 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚487085251053568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2215845674249957 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115393436 _step‚1197 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15391718783488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2200477418221568 $ _timestamp‚1.784285182803735e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15368238989029 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2227248940325376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134773984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31330502 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15367898439139 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15391377444352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15391718783488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250­ ¡#“¹-‚(:ˆ þèÒÖžó¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¨ º-‚(:ø þè҉Ń¥$ disk./.usageGB‚548.7866058349609 disk.drbd2165.out‚0 disk.rbd0.out‚0) proc.memory.percent‚17.22586154937744 cpu‚26.02593565561793 network.sent‚ 2.3104003e+07 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 494.2734375 disk.dm-1.in‚0 disk.rbd3.out‚0 network.recv‚ 3.294771e+06 disk.sda.out‚ 494.2734375 proc.cpu.threads‚17) disk./.usagePercent‚36.82952958021646# disk.nvme0c0n1.in‚ 144888.671875 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.sda.in‚0# disk.dm-0.out‚53701.56884765625( proc.memory.availableMB‚ 430.30859375# proc.memory.rssMB‚ 2822.28515625& disk.nvme0n1.out‚53701.56884765625 disk.sda3.in‚0! disk.nvme0n1.in‚ 144888.671875( disk.nvme0c0n1.out‚53701.56884765625 disk.sda2.out‚0! disk.rbd1.out‚115.07470703125 disk.rbd3.in‚0 disk.dm-0.in‚ 144888.671875# memory_percent‚97.3736047744751 disk.sda2.in‚0 disk.dm-1.out‚ 494.2734375=N^ý4»-/ _wandbruntime‚12375  _runtime‚123759î¹\'¼-‹N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15392731417600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15368909492861 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2215993195975663 % _timestamp‚1.7842851834152286e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15392390071296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2200623928893440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134783036 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2227397193989120 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15392390071296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15562961068544 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115401068 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250194097 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134783036 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250184104 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15392731417600 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115401068 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502681640880640 trainer/global_step‚14549 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1198 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31332282 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2227396835603456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115401070 _runtime‚12375.540993785 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250184104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56820375 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134793027 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15562862012928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154¢'½-N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115401070 _runtime‚12375.540993785 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250184104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56820375 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134793027 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15562862012928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15392731417600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15368909492861 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2215993195975663 % _timestamp‚1.7842851834152286e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15392390071296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2200623928893440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134783036 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2227397193989120 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15392390071296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15562961068544 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115401068 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250194097 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134783036 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250184104 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15392731417600 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115401068 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502681640880640 trainer/global_step‚14549 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1198 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31332282 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2227396835603456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088® `y©4¾-/ _wandbruntime‚12375  _runtime‚12375p¥¯¿¿-ˆ /loss/FBt2/detection_loss‚0.6039072215435621 %loss/total_loss‚7.063297450829257 % _timestamp‚1.7842851834181294e+09 _runtime‚12375.543462985 epoch‚36 _step‚1199 0loss/FRt1/assignment_loss‚1.4468539877168678 0loss/FRt2/assignment_loss‚0.8043399225229515 3loss/SRqqt1/assignment_loss‚0.37970778251862636 3loss/SRqqt2/assignment_loss‚0.39896067195770424 2loss/FBt1/assignment_loss‚0.004556238626411759 .loss/FBt1/detection_loss‚0.345154652300741 /loss/FRt2/detection_loss‚1.1856405130220464 1loss/SRqqt1/detection_loss‚0.7839674726999071 trainer/global_step‚14549 2loss/FBt2/assignment_loss‚0.000568342362951591 /loss/FRt1/detection_loss‚0.5228074464685671 1loss/SRqqt2/detection_loss‚0.5868331990889196YKuÀ- 0loss/FRt1/assignment_loss‚1.4468539877168678 0loss/FRt2/assignment_loss‚0.8043399225229515 3loss/SRqqt1/assignment_loss‚0.37970778251862636 3loss/SRqqt2/assignment_loss‚0.39896067195770424 2loss/FBt1/assignment_loss‚0.004556238626411759 .loss/FBt1/detection_loss‚0.345154652300741 /loss/FRt2/detection_loss‚1.1856405130220464 1loss/SRqqt1/detection_loss‚0.7839674726999071 trainer/global_step‚14549 2loss/FBt2/assignment_loss‚0.000568342362951591 /loss/FRt1/detection_loss‚0.5228074464685671 1loss/SRqqt2/detection_loss‚0.5868331990889196 /loss/FBt2/detection_loss‚0.6039072215435621 %loss/total_loss‚7.063297450829257 % _timestamp‚1.7842851834181294e+09 _runtime‚12375.543462985 epoch‚36 _step‚1199¯ ÜëۓÁ-‚(:ˆ èÒòÞ«¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚187.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÈ€.õÂ-‚(:ê èÒà÷¸¥ disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚144889.5$ memory_percent‚97.37372398376465# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17) disk./.usagePercent‚36.82956234918084 disk.sda.out‚ 494.5703125' proc.memory.availableMB‚ 430.2890625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚144889.5 disk.sda2.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚53712.13330078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚53712.13330078125 cpu‚26.025520661235408 network.sent‚ 2.3156519e+07 disk.rbd0.in‚0( disk.nvme0c0n1.out‚53712.13330078125 disk.sda.in‚0! disk.rbd1.out‚115.07470703125 disk.nvme0n1.in‚144889.5) proc.memory.percent‚17.22586154937744$ disk./.usageGB‚548.7870941162109 disk.sda3.out‚ 494.5703125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 494.5703125 network.recv‚ 3.299392e+06 disk.drbd2165.out‚0 disk.rbd3.out‚0yBIÃ-‚(:“ œèÒßÉ£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚258.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.762800000000006 gpu.0.smClock‚1410 gpu.0.memorM†rÿUyClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03dÜ$úÄ-‚(:ï œèÒøœ¦¥ network.sent‚ 2.3208881e+07 disk.rbd0.out‚0' proc.memory.availableMB‚ 430.7578125 disk.sda.out‚ 497.109375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0$ memory_percent‚97.37086296081543 disk.rbd1.in‚0.0625$ disk.nvme0n1.out‚53730.076171875# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17) disk./.usagePercent‚36.83026201777192 disk.sda.in‚0 disk.dm-1.out‚ 497.109375 disk.sda1.out‚0) proc.memory.percent‚17.22586154937744 network.recv‚ 3.303385e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 497.109375 disk.dm-1.in‚0 disk.dm-0.in‚ 144890.671875$ disk./.usageGB‚548.7975196838379# disk.nvme0c0n1.in‚ 144890.671875 disk.sda3.in‚0 disk.rbd3.in‚0! disk.dm-0.out‚53730.076171875 disk.drbd2165.in‚0 cpu‚26.02516918924558& disk.nvme0c0n1.out‚53730.076171875! disk.rbd1.out‚115.13720703125 disk.rbd3.out‚0! disk.nvme0n1.in‚ 144890.671875+E4Å-/ _wandbruntime‚12414  _runtime‚12414Ê´üJÊÆ-Ä & lr-AdamW/pg1‚8.316447427037192e-05 & lr-AdamW/pg2‚8.316447427037192e-05 trainer/global_step‚14599 % _timestamp‚1.7842852227559824e+09 _runtime‚12414.881589367 _step‚1200ŠQï:ÏÇ-É % _timestamp‚1.7842852227559824e+09 _runtime‚12414.881589367 _step‚1200 & lr-AdamW/pg1‚8.316447427037192e-05 & lr-AdamW/pg2‚8.316447427037192e-05 trainer/global_step‚14599° ™ ®4È-/ _wandbruntime‚12414  _runtime‚12414¢¥ã­(É-P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135238413 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2219219255984128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251003472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15442072946176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57006592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15612888690176 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2223221466741215 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7842852227570581e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15442414285312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚504369559057920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115775038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚1201 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115775036 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115775036 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135228436 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2234661328930304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚14599 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2234661687308800 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15418514926047 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504369672538624 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2207802951815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31432727 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15442414285312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31432991 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25573598 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2234661687308800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135238413 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25573601 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2219219273023488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2223221824330465 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251013451 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2207802968854528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115775038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488756688979968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15612983558656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2234661328930304 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15442072946176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2219219273023488Þ•§ (Ê-†P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚1201 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115775036 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115775036 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135228436 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2234661328930304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚14599 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2234661687308800 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15418514926047 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504369672538624 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2207802951815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31432727 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15442414285312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31432991 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25573598 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2234661687308800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135238413 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25573601 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2219219273023488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2223221824330465 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251013451 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2207802968854528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115775038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488756688979968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15612983558656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2234661328930304 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15442072946176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2219219273023488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135238413 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2219219255984128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251003472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15442072946176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57006592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15612888690176 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2223221466741215 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7842852227570581e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15442414285312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚504369559057920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115775038± ñ8ÏÃ4Ë-/ _wandbruntime‚12415  _runtime‚12415àµ]ž'Ì-‹N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2223368988466809 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15419866529771 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15419525979769 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25575343 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251030149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57010057 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15443426919424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135237488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135247479 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2207949479526400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57010324 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115782668 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2223369346056171 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2234809582586880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚14599 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504404117467136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚12415.494868326 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2219366514053120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15613999728640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251020156 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2219366497013760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25575346 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚488790117738496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488790099126272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2219366514053120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2207949462487040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15443085573120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57010324 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115782668 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2223369346056171 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2234809582586880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚14599 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504404117467136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚12415.494868326 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2219366514053120 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15613999728640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251020156 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2219366497013760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25575346 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚488790117738496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488790099126272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2219366514053120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2207949462487040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15443085573120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251030149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57010057 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15443426919424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135237488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135247479 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2207949479526400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25660848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25660851 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116156638 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251849503 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116156636 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135692865 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842852625838542e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2242074075913728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2226581307465728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31535746 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚490428108294144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15493109787136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15492768448000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135682888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1205 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135682888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2242074434292224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2230597616821723 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116156638 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2215128485408768 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57196597 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57196335 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57196597 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57196335 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TGDeviceStatsMonitor.on_trainlå–_batch_start/inactive_split.large_pool.freed‚25660848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25660851 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116156638 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251849503 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116156636 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135692865 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7842852625838542e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2242074075913728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2226581307465728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31535746 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚490428108294144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15493109787136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15492768448000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135682888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1205 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135682888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2242074434292224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2230597616821723 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116156638 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2215128485408768 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136µ §Lã?4Ý-/ _wandbruntime‚12455  _runtime‚12455«Q'ï'Þ-‹N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2226728548495360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚506126538014208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135691940 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2226728548495360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2230745138547317 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116164268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚490461555664896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135701931 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251856208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57200105 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31537512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 $ _timestamp‚1.784285263196699e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15493781074944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2242222687955968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2226728565534720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135691940 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2242222687955968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2242222329570304 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116164270 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2226728565534720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251866201 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25662596 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚12455.322561997 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15493781074944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25662593 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2215275013120000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251866201 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31537784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15470483016679 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1206 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15494122421248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135701931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚506126416152064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116164268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2215274996080640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103249920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15664879099392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116164270 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251856208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490461537052672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591ë¡'ß-N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2226728565534720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135691940 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2242222687955968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2242222329570304 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116164270 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2226728565534720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251866201 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25662596 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚12455.322561997 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15493781074944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25662593 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2215275013120000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251866201 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31537784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15470483016679 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1206 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15494122421248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135701931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚506126416152064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116164268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2215274996080640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103249920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15664879099392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116164270 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251856208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490461537052672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2226728548495360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚506126538014208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135691940 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2226728548495360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2230745138547317 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116164268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚490461555664896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135701931 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251856208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57200105 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31537512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 $ _timestamp‚1.784285263196699e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15493781074944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2242222687955968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360¶ *üš²4à-/ _wandbruntime‚12455  _runtime‚12455f³Ý á-‡ 1loss/SRqqt2/detection_loss‚0.6388676660620578 /loss/FBt1/detection_loss‚0.2642421564908004 _step‚1207 3loss/SRqqt1/assignment_loss‚0.33596070252390153 2loss/SRqqt2/assignment_loss‚0.4329822783695481 /loss/FRt1/detection_loss‚0.5307203655413378 /loss/FBt2/detection_loss‚0.6648978440373737 epoch‚36 _runtime‚12455.325298708 0loss/FRt2/assignment_loss‚0.7689279556492347 1loss/FBt1/assignment_loss‚0.00367543310897098 4loss/FBt2/assignment_loss‚0.00017647374092634647 /loss/FRt2/detection_loss‚1.2286682600971444 /loss/SRqqt1/detection_loss‚0.76185029693184 %loss/total_loss‚7.059798798906385 % _timestamp‚1.7842852631999502e+09 trainer/global_step‚14649 0loss/FRt1/assignment_loss‚1.4288293663532503<¡xSâ-Œ % _timestamp‚1.7842852631999502e+09 trainer/global_step‚14649 0loss/FRt1/assignment_loss‚1.4288293663532503 1loss/SRqqt2/detection_loss‚0.6388676660620578 /loss/FBt1/detection_loss‚0.2642421564908004 _step‚1207 3loss/SRqqt1/assignment_loss‚0.33596070252390153 2loss/SRqqt2/assignment_loss‚0.4329822783695481 /loss/FRt1/detection_loss‚0.5307203655413378 /loss/FBt2/detection_loss‚0.6648978440373737 epoch‚36 _runtime‚12455.325298708 0loss/FRt2/assignment_loss‚0.7689279556492347 1loss/FBt1/assignment_loss‚0.00367543310897098 4loss/FBt2/assignment_loss‚0.00017647374092634647 /loss/FRt2/detection_loss‚1.2286682600971444 /loss/SRqqt1/detection_loss‚0.76185029693184 %loss/total_loss‚7.059798798906385· (ÿi“ã-‚(:ˆ ØèÒØš±£ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ë·ä-‚(:û ØèÒ•…å¥" proc.memory.availableMB‚430.25$ disk./.usageGB‚548.7881660461426 disk.sda2.in‚0 disk.sda3.out‚ 504.18359375 disk.rbd3.in‚0% disk.nvme0n1.out‚53785.9541015625) proc.memory.percent‚17.22586154937744 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚144893.04296875" disk.dm-0.out‚53785.9541015625 disk.sda1.out‚0$ memory_percent‚97.37396240234375# proc.memory.rssMB‚ 2822.28515625 disk.sda1.in‚0 disk.rbd0.in‚0 network.recv‚ 3.318414e+06 disk.rbd1.in‚0.0625 disk.sda.out‚ 504.18359375 disk.dm-1.in‚0 disk.dm-1.out‚ 504.18359375! disk.rbd1.out‚115.32470703125' disk.nvme0c0n1.out‚53785.9541015625 cpu‚26.023687598923004 network.sent‚ 2.3287246e+07* disk./.usagePercent‚36.829634287297964 disk.sda2.out‚0# disk.nvme0n1.in‚144893.04296875 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144893.04296875 disk.sda.in‚073onžå-‚(:“ çèÒ’š¢ gpu.0.gpu‚90.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôýãæ-‚(:ú çèÒ­­¤¤ network.recv‚ 3.323011e+06$ disk./.usageGB‚548.7887344360352 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0! disk.rbd1.out‚115.32470703125 disk.sda1.out‚0# proc.memory.rssMB‚ 2822.28515625 disk.sda3.out‚ 504.67578125 disk.rbd1.in‚0.0625$ proc.memory.availableMB‚430.3125 cpu‚26.023306402825717 network.sent‚ 2.3350301e+07 disk.dm-0.in‚144893.04296875 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144893.04296875& disk.nvme0c0n1.out‚53791.455078125 disk.sda.in‚0 disk.sda.out‚ 504.67578125 disk.dm-1.in‚0) proc.memory.percent‚17.22586154937744 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚53791.455078125 disk.drbd2165.in‚0$ memory_percent‚97.37358093261719 proc.cpu.threads‚17* disk./.usagePercent‚36.829672432420566# disk.nvme0n1.in‚144893.04296875$ disk.nvme0n1.out‚53791.455078125 disk.drbd2165.out‚0 disk.dm-1.out‚ 504.67578125 disk.sda1.in‚0e;4ç-/ _wandbruntime‚12494  _runtime‚12494àMáÁÇè-Á % lr-AdamW/pg1‚8.02994542711744e-05 % lr-AdamW/pg2‚8.02994542711744e-05 trainer/global_step‚14699 % _timestamp‚1.7842853025248404e+09 _runtime‚12494.65046023 _step‚1208'| Ìé-Æ % lr-AdamW/pg1‚8.02994542711744e-05 % lr-AdamW/pg2‚8.02994542711744e-05 trainer/global_step‚14699 % _timestamp‚1.7842853025248404e+09 _runtime‚12494.65046023 _step‚1208¸ ”×^»4ê-/ _wandbruntime‚12494  _runtime‚12494óMlë-P XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57386435 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136147317 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2233943375986688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2237973766902231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842853025259109e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15715009912832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2249487181275648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15543463949824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2233943358947328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31638334 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492099546220544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252685555 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116538236 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31638071 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2249486822897152 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15714915044352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2222454036041728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136137340 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚12494.651681623 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116538236 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492099564832768 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15543805288960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15543805288960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507814574745600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2237974124491481 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15519747899863 trainer/global_step‚14699 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚5078144612648969Ü% (ì-†P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2233943375986688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2237973766902231 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842853025259109e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15715009912832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2249487181275648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15543463949824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2233943358947328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31638334 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492099546220544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252685555 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116538236 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31638071 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2249486822897152 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15714915044352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2222454036041728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136137340 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚12494.651681623 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116538236 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492099564832768 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15543805288960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15543805288960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507814574745600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2237974124491481 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15519747899863 trainer/global_step‚14699 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚507814461264896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57386435 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136147317 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976¹ ¸ää7“í-‚(:ˆ öèÒñÕµ¢ gpu.0.gpu‚81.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AaÂŒî-‚(:û öèÒèúû¥# disk.dm-0.out‚53804.33251953125$ memory_percent‚97.37424850463867) proc.memory.percent‚17.22586154937744 disk.rbd3.in‚0" disk.nvme0n1.in‚144893.7578125 disk.drbd2165.in‚0 disk.sda.out‚ 506.8671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚ 3.326515e+06& disk.nvme0n1.out‚53804.33251953125 disk.sda1.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 506.8671875& proc.memory.availableMB‚ 430.203125 disk.rbd3.out‚0 disk.dm-0.in‚144893.7578125 disk.sda1.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚144893.7578125 disk.sda3.out‚ 506.8671875) disk./.usagePercent‚36.82998092149932 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2822.28515625 cpu‚26.022972249484656 disk.drbd2165.out‚0 disk.sda.in‚0! disk.rbd1.out‚115.38720703125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚53804.33251953125 network.sent‚ 2.3355905e+07$ disk./.usageGB‚548.7933311462402ÁU¤4ï-/ _wandbruntime‚12495  _runtime‚12495wšQ'ð-‹N QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2222600546713600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15521099503587 trainer/global_step‚14699 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2234090617016320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136156383 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2249635434939392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136146392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252702253 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2249635076553728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15544476576768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚507849022864896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15520758953585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57389922 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136146392 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252692260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚12495.273119065 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252702253 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116545870 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507848915682816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2238121646217187 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15544817923072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31640338 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15716029273600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1210 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25749846 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116545868 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2249635434939392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136146392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252702253 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2249635076553728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.smallÌ­Qi#_pool.freed‚15544476576768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚507849022864896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15520758953585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57389922 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136146392 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252692260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚12495.273119065 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252702253 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116545870 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507848915682816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2238121646217187 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15544817923072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31640338 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15716029273600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1210 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25749846 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116545868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77º ºæ5¡4ò-/ _wandbruntime‚12495  _runtime‚12495Ò•Îöó-‹ 1loss/SRqqt2/detection_loss‚0.6935554837143242 0loss/FBt1/detection_loss‚0.33353558016192153 % _timestamp‚1.7842853031490042e+09 /loss/FRt1/assignment_loss‚1.491689881480292 1loss/SRqqt1/detection_loss‚0.9276799523909826 %loss/total_loss‚7.473190339434678 trainer/global_step‚14699 _step‚1211 2loss/FBt1/assignment_loss‚0.006457461194278454 4loss/FBt2/assignment_loss‚0.00032584284518784773 /loss/FRt1/detection_loss‚0.5034221936910181 /loss/FBt2/detection_loss‚0.7200272493889012 3loss/SRqqt1/assignment_loss‚0.36765714464511995 3loss/SRqqt2/assignment_loss‚0.43018918925169936 /loss/FRt2/detection_loss‚1.1855071152616632 epoch‚36 _runtime‚12495.274340017 0loss/FRt2/assignment_loss‚0.8131432454092905槦0ô- 3loss/SRqqt1/assignment_loss‚0.36765714464511995 3loss/SRqqt2/assignment_loss‚0.43018918925169936 /loss/FRt2/detection_loss‚1.1855071152616632 epoch‚36 _runtime‚12495.274340017 0loss/FRt2/assignment_loss‚0.8131432454092905 1loss/SRqqt2/detection_loss‚0.6935554837143242 0loss/FBt1/detection_loss‚0.33353558016192153 % _timestamp‚1.7842853031490042e+09 /loss/FRt1/assignment_loss‚1.491689881480292 1loss/SRqqt1/detection_loss‚0.9276799523909826 %loss/total_loss‚7.473190339434678 trainer/global_step‚14699 _step‚1211 2loss/FBt1/assignment_loss‚0.006457461194278454 4loss/FBt2/assignment_loss‚0.00032584284518784773 /loss/FRt1/detection_loss‚0.5034221936910181 /loss/FBt2/detection_loss‚0.7200272493889012» Ý##žõ-‚(:“ …‘èÒ×úŸ¥ gpu.0.gpu‚52.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.842600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hpgö-‚(:ý …‘èÒïས( disk.nvme0c0n1.out‚53836.74462890625 disk.rbd3.out‚0$ memory_percent‚97.37119674682617# disk./.usageGB‚548.790225982666& disk.nvme0n1.out‚53836.74462890625 disk.sda2.out‚0 cpu‚26.022623940524802 proc.cpu.threads‚17 network.recv‚ 3.331485e+06 disk.rbd0.in‚0 disk.sda.out‚ 510.41796875& proc.memory.availableMB‚ 430.703125! disk.rbd1.out‚115.38720703125# proc.memory.rssMB‚ 2822.28515625$ disk.nvme0c0n1.in‚144894.6171875 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.dm-0.in‚144894.6171875# disk.dm-0.out‚53836.74462890625 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 510.41796875) proc.memory.percent‚17.22586154937744) disk./.usagePercent‚36.82977253136645" disk.nvme0n1.in‚144894.6171875 disk.sda3.out‚ 510.41796875 disk.rbd3.in‚0 network.sent‚ 2.3420569e+07 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0J­ªñ“÷-‚(:ˆ ”‘èÒŽó㢠gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ž‘Òòø-‚(:ç ”‘èÒ½ÿ¥¥( proc.memory.availableMB‚ 430.15234375) disk./.usagePercent‚36.82976664319316 disk.sda.in‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚53849.05615234375 disk.sda1.out‚0( disk.nvme0c0n1.out‚53849.05615234375 disk.sda.out‚512# disk.nvme0n1.in‚144895.33203125# memory_percent‚97.3745584487915 disk.rbd3.out‚0 disk.dm-0.in‚144895.33203125 network.sent‚ 2.3460091e+07! disk.rbd1.out‚115.44970703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 3.335172e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚512 disk.sda1.in‚0# proc.memory.rssMB‚ 2822.28515625$ disk./.usageGB‚548.7901382446289 disk.rbd0.in‚0% disk.nvme0c0n1.in‚144895.33203125 disk.sda2.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.22586154937744 cpu‚26.022264046345704 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.out‚512 disk.dm-1.in‚0& disk.nvme0n1.out‚53849.05615234375rCÕÜ4ù-/ _wandbruntime‚12534  _runtime‚12534}*hÛÉú-à _runtime‚12534.63840714 _step‚1212 & lr-AdamW/pg1‚7.888193114855556e-05 & lr-AdamW/pg2‚7.888193114855556e-05 trainer/global_step‚14749 % _timestamp‚1.7842853425128024e+092Ö%Îû-È & lr-AdamW/pg1‚7.888193114855556e-05 & lr-AdamW/pg2‚7.888193114855556e-05 trainer/global_step‚14749 % _timestamp‚1.7842853425128024e+09 _runtime‚12534.63840714 _step‚1212¼ /ròˆ4ü-/ _wandbruntime‚12534  _runtime‚12534‚ÓÚ(ý-P F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2256899569880576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15766116469760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15766029989888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2256899569880576 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2241305427468288 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15594500790784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2229779569635328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57576883 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116919838 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15594159451648 trainer/global_step‚14749 % _timestamp‚1.7842853425138521e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2241305410428928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116919836 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136591792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253521607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚12534.639620262 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2241305410428928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25835348 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31741532 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253511628 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2256899928259072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601769 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚493770984146944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2256899928259072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116919838 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2245350274571989 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15594500790784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15594159451648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2229779552595968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15570704936661 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136591792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2241305427468288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116919836 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15570364386771x²¤” (þ-†P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253511628 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2256899928259072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601769 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚493770984146944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2256899928259072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116919838 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2245350274571989 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15594500790784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15594159451648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2229779552595968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15570704936661 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136591792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2241305427468288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116919836 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15570364386771 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2256899569880576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15766116469760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15766029989888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2256899569880576 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2241305427468288 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15594500790784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2229779569635328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57576883 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116919838 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15594159451648 trainer/global_step‚14749 % _timestamp‚1.7842853425138521e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2241305410428928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116919836 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136591792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253521607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚12534.639620262 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2241305410428928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25835348 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31741532 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088½ ÓÓå4ÿ-/ _wandbruntime‚12535  _runtime‚12535kéY¡€.‹N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31743504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493804431517696 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2245497438708333 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15595172078592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57580600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2241452651458560 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57580349 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚&E3p 15767060507136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2245497796297695 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚509571473412608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15595172078592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253528312 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15571375440493 _runtime‚12535.251299691 _step‚1214 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253538305 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2257048181922816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116927470 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253538305 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15595513424896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15595513424896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509571574303232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2229926063267840 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2241452668497920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31743504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493804431517696 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2245497438708333 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15595172078592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57580600 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2241452651458560 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57580349 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15767060507136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2245497796297695 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚509571473412608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15595172078592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253528312 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15571375440493 _runtime‚12535.251299691 _step‚1214 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253538305 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¾ àH–h4‚./ _wandbruntime‚12535  _runtime‚125352„ûƒ.Š epoch‚36 $ _timestamp‚1.784285343128236e+09 2loss/SRqqt2/assignment_loss‚0.4683565326053415 3loss/FBt1/assignment_loss‚0.0032452480790936384 1loss/SRqqt1/detection_loss‚0.7948196063995149 /loss/FBt2/detection_loss‚0.3630982488184236 trainer/global_step‚14749 _runtime‚12535.253760992 _step‚1215 1loss/SRqqt2/detection_loss‚0.6462636429304195 /loss/FBt1/detection_loss‚0.3871460929608249 0loss/FRt1/assignment_loss‚1.5919356843485033 0loss/FRt2/assignment_loss‚0.9418372628981607 2loss/SRqqt1/assignment_loss‚0.4230086544998056 4loss/FBt2/assignment_loss‚0.00015064425275966188 0loss/FRt1/detection_loss‚0.47569615558843714 /loss/FRt2/detection_loss‚1.1653689879773945 %loss/total_loss‚7.260926761358679@[™z„. 2loss/SRqqt2/assignment_loss‚0.4683565326053415 3loss/FBt1/assignment_loss‚0.0032452480790936384 1loss/SRqqt1/detection_loss‚0.7948196063995149 /loss/FBt2/detection_loss‚0.3630982488184236 trainer/global_step‚14749 _runtime‚12535.253760992 _step‚1215 1loss/SRqqt2/detection_loss‚0.6462636429304195 /loss/FBt1/detection_loss‚0.3871460929608249 0loss/FRt1/assignment_loss‚1.5919356843485033 0loss/FRt2/assignment_loss‚0.9418372628981607 2loss/SRqqt1/assignment_loss‚0.4230086544998056 4loss/FBt2/assignment_loss‚0.00015064425275966188 0loss/FRt1/detection_loss‚0.47569615558843714 /loss/FRt2/detection_loss‚1.1653689879773945 %loss/total_loss‚7.260926761358679 epoch‚36 $ _timestamp‚1.784285343128236e+09¿ –v s“….‚(:ˆ £‘èÒËÁþ¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lÁb¶†.‚(:ý £‘èÒÖ¥Œ¦) disk./.usagePercent‚36.82986469407878 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0$ memory_percent‚97.37193584442139 proc.cpu.threads‚17 network.sent‚ 2.3478309e+07 disk.drbd2165.in‚0 disk.sda.out‚ 512.765625! disk.rbd1.out‚115.44970703125 disk.rbd3.in‚0( disk.nvme0c0n1.out‚53863.88427734375$ disk./.usageGB‚548.7915992736816 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0# disk.dm-0.out‚53863.88427734375# disk.nvme0n1.in‚144896.16015625% disk.nvme0c0n1.in‚144896.16015625) proc.memory.percent‚17.22586154937744 disk.sda3.out‚ 512.765625 disk.dm-1.out‚ 512.765625 disk.dm-0.in‚144896.16015625 disk.sda2.out‚0# proc.memory.rssMB‚ 2822.28515625 cpu‚26.021856622548942 network.recv‚ 3.339023e+06 disk.sda3.in‚0& disk.nvme0n1.out‚53863.88427734375 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 430.58203125Î|h‡.‚(:’ ²‘èÒÎ𬣠gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áËwlˆ.‚(:û ²‘èÒåÄ£¥ network.sent‚ 2.3528506e+07 network.recv‚ 3.343607e+06 disk.rbd3.in‚0# disk.nvme0n1.out‚53867.94921875 cpu‚26.021536463993204 disk.rbd3.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2822.28515625 disk.dm-1.out‚ 513.71484375 disk.rbd1.in‚0.0625 disk.dm-0.in‚144896.16015625$ memory_percent‚97.37355709075928 disk.dm-1.in‚0 disk.dm-0.out‚53867.94921875 disk.sda1.in‚0 disk.sda.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.829676528541114 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144896.16015625% disk.nvme0c0n1.out‚53867.94921875 disk.sda.out‚ 513.71484375 disk.sda2.in‚0# disk.nvme0n1.in‚144896.16015625 disk.rbd0.in‚0 disk.sda3.out‚ 513.71484375) proc.memory.percent‚17.22586154937744$ disk./.usageGB‚548.7887954711914 disk.sda2.out‚0! disk.rbd1.out‚115.51220703125 disk.drbd2165.in‚0( proc.memory.availableMB‚ 430.31640625]™:“‰.‚(:ˆ Á‘èÒäÐ¥¢ gpu.0.gpu‚83.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚243.784' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.7568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜§úÔüŠ.‚(:ñ Á‘èÒ׿¶¤ disk.sda1.out‚0 disk.dm-1.out‚ 513.97265625 cpu‚26.021157837874547% disk.nvme0n1.out‚53876.6669921875 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 144896.875 disk.sda2.out‚0 disk.dm-0.in‚ 144896.875" disk.dm-0.out‚53876.6669921875 disk.nvme0n1.in‚ 144896.875 disk.sda2.in‚0 network.recv‚ 3.347115e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 513.97265625 disk.sda3.out‚ 513.97265625 disk.rbd3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 430.06640625$ disk./.usageGB‚548.7846984863281' disk.nvme0c0n1.out‚53876.6669921875# proc.memory.rssMB‚ 2822.28515625) disk./.usagePercent‚36.82940157644934 disk.drbd2165.out‚0 disk.dm-1.in‚0) proc.memory.percent‚17.22586154937744 proc.cpu.threads‚17 network.sent‚ 2.3533568e+07 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚115.51220703125$ memory_percent‚97.37508296966553¢Ýú•4‹./ _wandbruntime‚12574  _runtime‚12574 &äîÉŒ.à & lr-AdamW/pg1‚7.747452088496603e-05 & lr-AdamW/pg2‚7.747452088496603e-05 trainer/global_step‚14799 % _timestamp‚1.7842853821859105e+09 _runtime‚12574.31149378 _step‚1216ï YÀÎ.È _runtime‚12574.31149378 _step‚1216 & lr-AdamW/pg1‚7.747452088496603e-05 & lr-AdamW/pg2‚7.747452088496603e-05 trainer/global_step‚14799 % _timestamp‚1.7842853821859105e+09À ÿìÝÁ4Ž./ _wandbruntime‚12574  _runtime‚12574e~äZå.P G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137056221 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15645196292608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2264312316864000 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2264312316864000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15645196292608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _runtime‚12574.312709064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495442422073344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15644854953472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2252726067063247 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842853821869357e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117301438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57767559 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15621321423569 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117301436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25922601 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2248667478949888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137056221 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31844707 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2252726424652497 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2248667478949888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137046244 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57767305 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495442440685568 trainer/global_step‚14799 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.f|^ ." reed‚2248667461910528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15644854953472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254357659 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2248667461910528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511259553667072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15817131593728 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2237105103228928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2264312675242496 _step‚1217 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137046244 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31844958 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117301436 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0#Q;D (.†P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2248667461910528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511259553667072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15817131593728 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2237105103228928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2264312675242496 _step‚1217 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137046244 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31844958 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117301436 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137056221 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15645196292608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2264312316864000 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2264312316864000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15645196292608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _runtime‚12574.312709064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495442422073344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15644854953472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2252726067063247 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842853821869357e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117301438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57767559 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15621321423569 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117301436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25922601 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2248667478949888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137056221 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31844707 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2252726424652497 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2248667478949888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137046244 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57767305 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495442440685568 trainer/global_step‚14799 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2248667461910528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15644854953472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254357659 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508Á ã¾@(4‘./ _wandbruntime‚12574  _runtime‚12574rU|â'’.‹N BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137055296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31847027 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57771091 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25924343 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25924346 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15818145841664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _step‚1218 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254374357 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117309068 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117309070 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495475869444096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117309068 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2252873946378203 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚12574.924677904 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15645867580416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511294110147072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31846748 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495475850831872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15622332477403 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254374357 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2252873588788841 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15818240702976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2237251596861440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2248814702940160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254364364 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117309070 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137065287 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2248814719979520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2237251613900800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15621991927401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2248814702940160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15645867580416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15646208926720 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137055296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31847027 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57771091 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25924343 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25924346 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15818145841664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _step‚1218 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254374357 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117309068 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117309070 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495475869444096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117309068 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2252873946378203 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚12574.924677904 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15645867580416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511294110147072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31846748 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495475850831872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15622332477403 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254374357 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2252873588788841 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15818240702976 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2237251596861440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2248814702940160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254364364 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117309070 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137065287 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2248814719979520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2237251613900800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15621991927401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2248814702940160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ¾g|4”./ _wandbruntime‚12574  _runtime‚12574fEÿ•.‹ 2loss/SRqg'8ÿqt2/assignment_loss‚0.4434829136047678 3loss/FBt1/assignment_loss‚0.0003957351072425675 0loss/FRt1/detection_loss‚0.46501130222912085 /loss/FRt2/detection_loss‚1.1662076873036957 1loss/SRqqt1/detection_loss‚0.8466874288524279 1loss/SRqqt2/detection_loss‚0.6101227394511052 _runtime‚12574.927505673 0loss/FRt1/assignment_loss‚1.4888351404780673 /loss/FRt2/assignment_loss‚0.811773575815866 3loss/FBt2/assignment_loss‚0.0003497659768773223 epoch‚36 trainer/global_step‚14799 _step‚1219 0loss/FBt2/detection_loss‚0.43435262513330913 % _timestamp‚1.7842853828021684e+09 0loss/FBt1/detection_loss‚0.32028271419455956 %loss/total_loss‚6.947520483937131 2loss/SRqqt1/assignment_loss‚0.3600188557900915Ù?i–. 0loss/FBt1/detection_loss‚0.32028271419455956 %loss/total_loss‚6.947520483937131 2loss/SRqqt1/assignment_loss‚0.3600188557900915 2loss/SRqqt2/assignment_loss‚0.4434829136047678 3loss/FBt1/assignment_loss‚0.0003957351072425675 0loss/FRt1/detection_loss‚0.46501130222912085 /loss/FRt2/detection_loss‚1.1662076873036957 1loss/SRqqt1/detection_loss‚0.8466874288524279 1loss/SRqqt2/detection_loss‚0.6101227394511052 _runtime‚12574.927505673 0loss/FRt1/assignment_loss‚1.4888351404780673 /loss/FRt2/assignment_loss‚0.811773575815866 3loss/FBt2/assignment_loss‚0.0003497659768773223 epoch‚36 trainer/global_step‚14799 _step‚1219 0loss/FBt2/detection_loss‚0.43435262513330913 % _timestamp‚1.7842853828021684e+09à uå•@œ—.‚(:‘ БèÒ– Í¤ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ"Oº˜.‚(:ý БèÒ‹Ì⥠disk.rbd0.out‚0 disk.dm-1.out‚ 514.30078125# proc.memory.rssMB‚ 2822.28515625! disk.nvme0n1.in‚ 144897.703125 disk.rbd1.in‚0.0625$ memory_percent‚97.37207889556885) proc.memory.percent‚17.22586154937744& disk.nvme0n1.out‚53883.90185546875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 514.30078125 disk.sda3.out‚ 514.30078125! disk.rbd1.out‚115.63720703125 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 2.3538637e+07$ disk./.usageGB‚548.7891082763672( disk.nvme0c0n1.out‚53883.90185546875 cpu‚26.023917811566616 network.recv‚ 3.350349e+06) disk./.usagePercent‚36.82969752115892 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 144897.703125( proc.memory.availableMB‚ 430.55859375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 144897.703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0# disk.dm-0.out‚53883.90185546875ÃÌm…4™./ _wandbruntime‚12592  _runtime‚12592,Ëd‚=š.üz 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979740977287292 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3653991222381592 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6126993298530579 _step‚1220 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960803985595703 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7514928579330444 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8012187480926514 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961787462234497 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7383220195770264 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9482102394104004 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_1_of_1‚0.33620181679725647 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.328125 trainer/global_step‚14799 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6377994418144226 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953734874725342 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6073905229568481 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021714212372899055 *jet/accuracy_2_of_3‚0.5797528028488159 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7393612861633301 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9930327534675598 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970018267631531 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22730691730976105 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7781000733375549 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9852258563041687 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.259768545627594 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21715736389160156 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2862148880958557 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38195541501045227 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0194742102175951 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8478623628616333 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46510690450668335 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7226184010505676 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6704837083816528 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.013199817389249802 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )loss/val_total_loss‚7.967518455205483 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24687151610851288 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7036892771720886 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2189357876777649 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6717819571495056 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08689948171377182 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2755661904811859 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6303359270095825 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9926129579544067 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7685930132865906 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2761322259902954 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8790146708488464 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,jet/accuracy_4_of_4‚0.011098379269242287 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2278670370578766 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22879146039485931 (particle/accuracy‚0.8436026573181152 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33164989948272705 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_3_of_3‚0.059215251356363297 *jet/accuracy_2_of_4‚0.8093563914299011 +particle/specificity‚0.9044818878173828 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6476933360099792 /particle/accuracy_3_of_4‚0.9979296326637268 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9911797046661377 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.653652548789978 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24133960902690887 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 epoch‚36 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5452325344085693 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26521897315979004 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.67060786485672 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2858301103115082 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045885879546403885 particle/accuracy_1_of_2‚1 % _timestamp‚1.7842854001177993e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25777652859687805 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20485880970954895 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9006174206733704 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.013199817389249802 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2755872309207916 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019332218915224075 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6737326383590698 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23877565562725067 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6193585991859436 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.578908383846283 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6306335926055908 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33995628356933594 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21361619234085083 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24904754757881165 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6648858785629272 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.607389509677887 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36513614654541016 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021525682881474495 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07565755397081375 *jet/accuracy_1_of_3‚0.9284268617630005 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6149777173995972 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6398102641105652 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2755661904811859 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6149777173995972 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8790146708488464 *jet/accuracy_1_of_4‚0.9804345369338989 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6360457539558411 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.6209791302680969 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3793060779571533 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6316255927085876 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7713650465011597 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28623178601264954 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6668036580085754 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039540473371744156 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49789002537727356 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996785581111908 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5792072415351868 +jet/accuracy_3_of_4‚0.08054506778717041 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7672387957572937 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518516421318054 particle/accuracy_2_of_2‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7313094139099121 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5109086036682129 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1341666728258133 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6619347333908081 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30633577704429626 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5990399122238159 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49442777037620544 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_2_of_3‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9932169914245605 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9910410642623901 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7453137040138245 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9915816187858582 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952129125595093 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9918553233146667 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2856771945953369 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7405896186828613 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997230052947998 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6321998238563538 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.927219569683075 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_1_of_2‚0.7988097071647644 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9180601239204407 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5108333826065063 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8572075963020325 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6372039318084717 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 _runtime‚12592.244007286 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047776561230421066 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5645753741264343 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5294841527938843 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04007946699857712 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7270854115486145 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26703429222106934 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5975770950317383 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.641376256942749 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20849153399467468 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7954437732696533 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 *jet/accuracy_2_of_2‚0.1616787165403366 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7002564072608948 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24337327480316162 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.271679550409317n~`6›.{ :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5452325344085693 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26521897315979004 7"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.67060786485672 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2858301103115082 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045885879546403885 particle/accuracy_1_of_2‚1 % _timestamp‚1.7842854001177993e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25777652859687805 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20485880970954895 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9006174206733704 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.013199817389249802 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2755872309207916 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019332218915224075 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6737326383590698 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23877565562725067 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6193585991859436 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.578908383846283 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6306335926055908 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33995628356933594 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21361619234085083 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24904754757881165 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6648858785629272 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.607389509677887 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36513614654541016 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021525682881474495 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07565755397081375 *jet/accuracy_1_of_3‚0.9284268617630005 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6149777173995972 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6398102641105652 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2755661904811859 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6149777173995972 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8790146708488464 *jet/accuracy_1_of_4‚0.9804345369338989 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6360457539558411 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.6209791302680969 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3793060779571533 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6316255927085876 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7713650465011597 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28623178601264954 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6668036580085754 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039540473371744156 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49789002537727356 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996785581111908 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5792072415351868 +jet/accuracy_3_of_4‚0.08054506778717041 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7672387957572937 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518516421318054 particle/accuracy_2_of_2‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7313094139099121 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5109086036682129 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1341666728258133 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6619347333908081 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30633577704429626 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5990399122238159 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49442777037620544 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_2_of_3‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9932169914245605 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9910410642623901 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7453137040138245 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9915816187858582 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952129125595093 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9918553233146667 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2856771945953369 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7405896186828613 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997230052947998 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6321998238563538 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.927219569683075 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 *jet/accuracy_1_of_2‚0.7988097071647644 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9180601239204407 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5108333826065063 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8572075963020325 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6372039318084717 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 _runtime‚12592.244007286 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047776561230421066 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5645753741264343 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5294841527938843 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04007946699857712 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7270854115486145 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26703429222106934 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5975770950317383 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.641376256942749 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20849153399467468 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7954437732696533 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 *jet/accuracy_2_of_2‚0.1616787165403366 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7002564072608948 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_4‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24337327480316162 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.271679550409317 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979740977287292 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3653991222381592 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6126993298530579 _step‚1220 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960803985595703 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7514928579330444 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8012187480926514 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961787462234497 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7383220195770264 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9482102394104004 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_1_of_1‚0.33620181679725647 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.328125 trainer/global_step‚14799 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6377994418144226 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953734874725342 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6073905229568481 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021714212372899055 *jet/accuracy_2_of_3‚0.5797528028488159 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7393612861633301 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9930327534675598 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970018267631531 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22730691730976105 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7781000733375549 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9852258563041687 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.259768545627594 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21715736389160156 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2862148880958557 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38195541501045227 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0194742102175951 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8478623628616333 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46510690450668335 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7226184010505676 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6704837083816528 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.013199817389249802 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )loss/val_total_loss‚7.967518455205483 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24687151610851288 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7036892771720886 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2189357876777649 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6717819571495056 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08689948171377182 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27556619048ø4'11859 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6303359270095825 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9926129579544067 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7685930132865906 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2761322259902954 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8790146708488464 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,jet/accuracy_4_of_4‚0.011098379269242287 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2278670370578766 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22879146039485931 (particle/accuracy‚0.8436026573181152 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33164989948272705 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_3_of_3‚0.059215251356363297 *jet/accuracy_2_of_4‚0.8093563914299011 +particle/specificity‚0.9044818878173828 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6476933360099792 /particle/accuracy_3_of_4‚0.9979296326637268 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9911797046661377 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.653652548789978 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24133960902690887 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 epoch‚36Ä ¢…Ì©œ.‚(:’ ß‘èÒªÿ £ gpu.0.gpu‚18.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚247.509' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.501799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿©‚.‚(:ø ß‘èÒ¹ï¥ network.recv‚ 3.355081e+06# disk.nvme0n1.out‚53892.80859375 disk.drbd2165.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 429.7890625 disk.dm-0.in‚144898.41796875# disk.nvme0n1.in‚144898.41796875% disk.nvme0c0n1.out‚53892.80859375 disk.sda.out‚ 514.57421875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 network.sent‚ 2.3601795e+07 disk.sda2.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.37677574157715( disk./.usagePercent‚36.8296936810459$ disk./.usageGB‚548.7890510559082 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.out‚ 514.57421875) proc.memory.percent‚17.22586154937744 disk.dm-0.out‚53892.80859375 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144898.41796875 proc.cpu.threads‚17 disk.rbd0.out‚0# proc.memory.rssMB‚ 2822.28515625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 514.57421875! disk.rbd1.out‚115.63720703125 cpu‚26.025829140795512®ëO{ž.‚(:’ î‘èÒŸ¾²¢ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.52380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07W¸è Ÿ.‚(:€ î‘èÒÄÚϤ disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0 cpu‚26.025465489075888 disk.drbd2165.in‚0 disk.dm-0.in‚144899.24609375) proc.memory.percent‚17.22586154937744 disk.rbd1.in‚0.0625( disk.nvme0c0n1.out‚53898.88232421875$ disk./.usageGB‚548.7855758666992 disk.dm-1.in‚0$ memory_percent‚97.37911224365234% proc.memory.availableMB‚ 429.40625# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17 network.recv‚ 3.359207e+06# disk.nvme0n1.in‚144899.24609375 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 514.73828125 disk.sda1.in‚0 disk.sda3.out‚ 514.73828125# disk.dm-0.out‚53898.88232421875 disk.sda.in‚0 disk.rbd3.in‚0% disk.nvme0c0n1.in‚144899.24609375 network.sent‚ 2.3659238e+07) disk./.usagePercent‚36.82946045818221 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 514.73828125 disk.sda2.in‚0! disk.rbd1.out‚115.69970703125& disk.nvme0n1.out‚53898.88232421875¿(Çž .‚(:“ ý‘èÒáˆÇ¢ gpu.0.gpu‚76.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]Y•Êõ¡.‚(:ê ý‘èÒÞÏ›¤( proc.memory.availableMB‚ 429.40234375$ disk./.usageGB‚548.7836151123047 disk.dm-1.out‚ 515.03515625 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 network.recv‚ 3.362606e+06 disk.sda.in‚0 disk.sda.out‚ 515.03515625 network.sent‚ 2.3664362e+07 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 disk.dm-1.in‚0! disk.rbd1.out‚115.69970703125 disk.dm-0.out‚ 53905.625 disk.nvme0n1.out‚ 53905.625$ memory_percent‚97.37913608551025 proc.cpu.threads‚17) disk./.usagePercent‚36.82932887030962 disk.nvme0c0n1.out‚ 53905.625% disk.nvme0c0n1.in‚144899.24609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 515.03515625# disk.nvme0n1.in‚144899.24609375 disk.sda3.in‚0 disk.dm-0.in‚144899.24609375 disk.rbd0.in‚0 cpu‚26.02514127246766Ð I"4¢./ _wandbruntime‚12631  _runtime‚12631lo:DÉ£.à % _timestamp‚1.7842854393343022e+09 _runtime‚12631.45991059 _step‚1221 & lr-AdamW/pg1‚7.607731387573691e-05 & lr-AdamW/pg2‚7.607731387573691e-05 trainer/global_step‚14849·C™Î¤.È trainer/global_step‚14849 % _timestamp‚1.7842854393343022e+09 _runtime‚12631.45991059 _step‚1221 & lr-AdamW/pg1‚7.607731387573691e-05 & lr-AdamW/pg2‚7.607731387573691e-05Å êÜb94¥./ _wandbruntime‚12631  _runtime‚12631ß)f(¦.ÿO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117803702 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15713283660800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2246339282997248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15713624999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255384699 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2262028600083064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137580995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137580995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2273672363703296 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2257959097081856 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚513016425646592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚497129772099584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15886540066304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2273672363703296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2262028957672314 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2257959097081856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26012371 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1222 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26012368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31978090 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15713624999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117803704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2257959080042496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2246339300036608 trainer/global_step‚14849 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚513016312165888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 $ _timestamp‚1.784285439335332e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚12631.46437215 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2273672722081792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57990461uK~ (§.„P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2246339282997248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15713624999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255384699 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2262028600083064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137580995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137580995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2273672363703296 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2257959097081856 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚513016425646592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚497129772099584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15886540066304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2273672363703296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2262028957672314 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2257959097081856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26012371 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1222 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26012368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31978090 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15713624999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117803704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2257959080042496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2246339300036608 trainer/global_step‚14849 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚513016312165888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 $ _timestamp‚1.784285439335332e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚12631.46437215 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2273672722081792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57990461 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117803702 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15713283660800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480Æ 3½h4¨./ _wandbruntime‚12632  _runtime‚12632—À ÖS©.ŠN ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255401397 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2258106321072128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31980151 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1223 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚513050872184832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137590061 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31979885 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15887652714496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15887568338944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2273820975745536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚513050769197056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2262176121808658 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117811336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2258106338111488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15714637634048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚497163200858112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2273820617359872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117811334 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]JDeviceStatsMonifopè½tor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15714637634048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26014116 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚12632.073412278 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117811336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137580070 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15714296287744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137580070 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57994267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497163219470336 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2246485793669120 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57993998 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15714296287744 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2262176479398020 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2258106338111488 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2258106321072128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15690668689540 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255391404 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26014113 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2246485810708480 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117811334R™~'ª.N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2262176479398020 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2258106338111488 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2258106321072128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15690668689540 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255391404 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26014113 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2246485810708480 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117811334 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255401397 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2258106321072128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31980151 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1223 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚513050872184832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137590061 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31979885 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15887652714496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15887568338944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2273820975745536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚513050769197056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2262176121808658 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117811336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2258106338111488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15714637634048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚497163200858112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2273820617359872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117811334 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15714637634048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26014116 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚12632.073412278 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117811336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137580070 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15714296287744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137580070 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57994267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497163219470336 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2246485793669120 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57993998 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15714296287744Ç Ç¯"í4«./ _wandbruntime‚12632  _runtime‚12632íæwd ¬.† 1loss/SRqqt1/detection_loss‚0.8060075086340758 trainer/global_step‚14849 /loss/FBt2/detection_loss‚0.2677602539339137 % _timestamp‚1.7842854399503868e+09 _step‚1224 2loss/SRqqt1/assignment_loss‚0.3924178277880142 2loss/SRqqt2/assignment_loss‚0.5033986827969354 /loss/FRt2/detection_loss‚1.2865213295613978 /loss/FBt1/detection_loss‚0.4533786483496174 %loss/total_loss‚7.024505830325131 epoch‚37 _runtime‚12632.07572397 0loss/FRt1/assignment_loss‚1.4792059281648233 /loss/FRt1/detection_loss‚0.5203326745687804 1loss/SRqqt2/detection_loss‚0.5694472996926567 0loss/FRt2/assignment_loss‚0.7376558279091457 2loss/FBt1/assignment_loss‚0.004455641890504359 2loss/FBt2/assignment_loss‚0.003924207035266681¦‹ú ­.‹ 2loss/SRqqt1/assignment_loss‚0.3924178277880142 2loss/SRqqt2/assignment_loss‚0.5033986827969354 /loss/FRt2/detection_loss‚1.2865213295613978 /loss/FBt1/detection_loss‚0.4533786483496174 %loss/total_loss‚7.024505830325131 epoch‚37 _runtime‚12632.07572397 0loss/FRt1/assignment_loss‚1.4792059281648233 /loss/FRt1/detection_loss‚0.5203326745687804 1loss/SRqqt2/detection_loss‚0.5694472996926567 0loss/FRt2/assignment_loss‚0.7376558279091457 2loss/FBt1/assignment_loss‚0.004455641890504359 2loss/FBt2/assignment_loss‚0.003924207035266681 1loss/SRqqt1/detection_loss‚0.8060075086340758 trainer/global_step‚14849 /loss/FBt2/detection_loss‚0.2677602539339137 % _timestamp‚1.7842854399503868e+09 _step‚1224È ߧ.“®.‚(:ˆ Œ’èÒò—Ÿ¤ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gžX ¯.‚(:€ Œ’èÒ–âÿ¥ disk.sda.out‚ 517.93359375) proc.memory.percent‚17.22586154937744" disk.nvme0n1.in‚144899.9609375 disk.drbd2165.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0! disk.rbd1.out‚115.76220703125& disk.nvme0n1.out‚53929.38916015625 disk.dm-1.out‚ 517.93359375 disk.rbd3.in‚0 disk.sda1.in‚0$ memory_percent‚97.37610816955566' proc.memory.availableMB‚ 429.8984375$ disk./.usageGB‚548.7896347045898 disk.rbd3.out‚0 disk.dm-0.in‚144899.9609375* disk./.usagePercent‚36.829732850198646# disk.dm-0.out‚53929.38916015625 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚144899.9609375( disk.nvme0c0n1.out‚53929.38916015625 disk.dm-1.in‚0 network.recv‚ 3.365885e+06 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 517.93359375 cpu‚26.024729618367893 network.sent‚ 2.3669431e+07 disk.rbd1.in‚0.0625 disk.sda1.out‚0M3ÜÛž°.‚(:“ ›’èÒ£·µ£ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rLY| ±.‚(:€ ›’èҼ˓¤* disk./.usagePercent‚36.829523436035636$ disk.nvme0c0n1.in‚144900.7890625 disk.rbd3.in‚0" disk.nvme0n1.in‚144900.7890625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚97.37725257873535 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.22586154937744 disk.sda2.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625& disk.nvme0n1.out‚53945.17626953125 disk.sda3.in‚0 disk.dm-1.out‚ 520.60546875! disk.rbd1.out‚115.76220703125 network.sent‚ 2.3731632e+07 disk.sda.out‚ 520.60546875 disk.rbd3.out‚0' proc.memory.availableMB‚ 429.7109375# proc.memory.rssMB‚ 2822.28515625$ disk./.usageGB‚548.7865142822266 disk.sda3.out‚ 520.60546875 disk.dm-0.in‚144900.7890625 cpu‚26.024373827388068 network.recv‚ 3.369601e+06 disk.rbd0.out‚0 disk.sda.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚53945.17626953125# disk.dm-0.out‚53945.17626953125/[p4²./ _wandbruntime‚12671  _runtime‚12671»F!¬Ê³.Ä trainer/global_step‚14899 % _timestamp‚1.7842854792621496e+09 _runtime‚12671.387738609 _step‚1225 & lr-AdamW/pg1‚7.469039986086364e-05 & lr-AdamW/pg2‚7.469039986086364e-0594òÏ´.É % _timestamp‚1.7842854792621496e+09 _runtime‚12671.387738609 _step‚1225 & lr-AdamW/pg1‚7.469039986086364e-05 & lr-AdamW/pg2‚7.469039986086364e-05 trainer/global_step‚14899É ‹[4µ./ _wandbruntime‚12671  _runtime‚12671/›|¼(¶.P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514739249701888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2281085469065216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26099618 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842854792631857e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2281085469065216 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15739933572724 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26099621 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15763979162624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498801210025984 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15938021063680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2269404750163572 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2265321131524096 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118185304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2281085110686720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58181385 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚12671.388942632 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1226 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2265321148563456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2253664816590848 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138035447 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256220751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256210772 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15764320501760 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2265321131524096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514739142512640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15740274122614 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2253664833630208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118185304 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2281085110686720 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2265321148563456 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58181648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498801228638208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15764320501760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138025470 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32082027 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118185302 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚14899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118185302 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138035447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32081767 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888ÉÒq&% ·.†P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2281085469065216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26099618 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842854792631857e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2281085469065216 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15739933572724 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26099621 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15763979162624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498801210025984 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15938021063680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2269404750163572 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2265321131524096 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118185304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2281085110686720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58181385 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtS,Ö˜çime‚12671.388942632 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1226 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2265321148563456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2253664816590848 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138035447 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256220751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256210772 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15764320501760 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2265321131524096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514739142512640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15740274122614 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2253664833630208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118185304 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2281085110686720 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2265321148563456 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58181648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498801228638208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15764320501760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138025470 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32082027 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118185302 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚14899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118185302 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138035447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32081767 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514739249701888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600Ê Ìê)E4¸./ _wandbruntime‚12672  _runtime‚12672ìœïŽ'¹.‹N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15764991789568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498834638784512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2253811344302080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256237449 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32084061 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118192934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138034522 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256237449 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138044513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2265468372553728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2269552271889166 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2269552629478528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15741285176448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15939045020160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256227456 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚514773702416896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32083787 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138044513 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15938958547456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚12672.000719202 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15765333135872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118192936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2253811327262720 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2265468389593088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1227 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118192936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15765333135872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498834657396736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15764991789568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138034522 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2265468372553728 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2281233722728960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58185150 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15740944626446 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2281233722728960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118192934 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32083787 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138044513 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15938958547456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚12672.000719202 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15765333135872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118192936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2253811327262720 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2265468389593088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1227 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118192936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15765333135872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498834657396736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15764991789568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138034522 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2265468372553728 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2281233722728960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58185150 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15740944626446 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2281233722728960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118192934 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256237449 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32084061 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118192934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138034522 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256237449 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138044513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2265468372553728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2269552271889166 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2269552629478528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15741285176448Ë 8øÀ4»./ _wandbruntime‚12672  _runtime‚12672¬æ¼.ˆ 1loss/SRqqt2/detection_loss‚0.6200744079404487 % _timestamp‚1.7842854798777308e+09 _step‚1228 %loss/total_loss‚7.332183596580149 /loss/FRt1/assignment_loss‚1.462447925266129 0loss/FRt2/assignment_loss‚0.7933996469050915 3loss/SRqqt1/assignment_loss‚0.39285865412313087 /loss/FRt2/detection_loss‚1.2101775353153021 1loss/SRqqt1/detection_loss‚0.9575321405227522 /loss/FBt1/detection_loss‚0.4957295967392232 0loss/FBt2/detection_loss‚0.44935721665387973 2loss/SRqqt2/assignment_loss‚0.3806289303943113 3loss/FBt2/assignment_loss‚0.0005632384045360019 /loss/FRt1/detection_loss‚0.5357187927285805 epoch‚37 trainer/global_step‚14899 1loss/FBt1/assignment_loss‚0.03369551158676351 _runtime‚12672.003067785o`¾È½. %loss/total_loss‚7.332183596580149 /loss/FRt1/assignment_loss‚1.462447925266129 0loss/FRt2/assignment_loss‚0.7933996469050915 3loss/SRqqt1/assignment_loss‚0.39285865412313087 /loss/FRt2/detection_loss‚1.2101775353153021 1loss/SRqqt1/detection_loss‚0.9575321405227522 /loss/FBt1/detection_loss‚0.4957295967392232 0loss/FBt2/detection_loss‚0.44935721665387973 2loss/SRqqt2/assignment_loss‚0.3806289303943113 3loss/FBt2/assignment_loss‚0.0005632384045360019 /loss/FRt1/detection_loss‚0.5357187927285805 epoch‚37 trainer/global_step‚14899 1loss/FBt1/assignment_loss‚0.03369551158676351 _runtime‚12672.003067785 1loss/SRqqt2/detection_loss‚0.6200744079404487 % _timestamp‚1.7842854798777308e+09 _step‚1228Ì ï`ñ)œ¾.‚(:‘ ª’èÒ‡ÿ¤ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚247.268' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.453599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬µ“¿.‚(:ý ª’èÒÚö‚¥ disk.sda3.in‚0 disk.sda3.out‚ 521.671875! disk.rbd1.out‚115.82470703125& disk.nvme0n1.out‚53952.72607421875 disk.sda.in‚0( proc.memory.availableMB‚ 429.72265625) proc.memory.percent‚17.22586154937744 proc.cpu.threads‚17 network.recv‚ 3.372954e+06 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2822.28515625 network.sent‚ 2.3736585e+07 disk.dm-1.out‚ 521.671875 disk.rbd1.in‚0.0625 disk.dm-0.in‚144901.50390625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.out‚ 521.671875) disk./.usagePercent‚36.82932989433975 disk.dm-1.in‚0 disk.sda2.out‚0# disk.dm-0.out‚53952.72607421875# disk.nvme0n1.in‚144901.50390625 disk.drbd2165.out‚0 disk.rbd0.out‚0$ memory_percent‚97.37718105316162 disk.rbd3.in‚0( disk.nvme0c0n1.out‚53952.72607421875$ disk./.usageGB‚548.7836303710938% disk.nvme0c0n1.in‚144901.50390625 cpu‚26.024046541130666˜ÂÆ£œÀ.‚(:‘ ¹’èÒ€ÙË£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šLµýÁ.‚(:û ¹’èÒ€ÏÖ¤ disk.sda1.out‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625* disk./.usagePercent‚36.829798132119876 disk.rbd0.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2822.28515625$ disk.nvme0n1.out‚53959.755859375 disk.sda1.in‚0 disk.sda2.out‚0! disk.rbd1.out‚115.82470703125$ memory_percent‚97.37682342529297 proc.cpu.threads‚17$ disk./.usageGB‚548.7906074523926 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 522.16796875! disk.dm-0.out‚53959.755859375 network.sent‚ 2.3799628e+07 network.recv‚ 3.377713e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144901.58984375 disk.sda.out‚ 522.16796875 disk.sda3.in‚0& disk.nvme0c0n1.out‚53959.755859375 disk.rbd3.in‚0) proc.memory.percent‚17.22586154937744 disk.drbd2165.in‚0% proc.memory.availableMB‚ 429.78125 disk.sda3.out‚ 522.16796875 disk.dm-0.in‚144901.58984375 cpu‚26.023685996288954# disk.nvme0n1.in‚144901.58984375C/Ø“Â.‚(:ˆ È’èÒ‰«Ø¢ gpu.0.gpu‚55.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oŒºD5Ã.‚(:„ È’èÒø‰„¥ disk.drbd2165.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0* disk./.usagePercent‚36.829806068353435$ disk./.usageGB‚548.7907257080078 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚115.88720703125 cpu‚26.02334ƒÀ>:Ú2918325504 disk.drbd2165.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚53966.72216796875 disk.sda1.in‚0( disk.nvme0c0n1.out‚53966.72216796875$ memory_percent‚97.37660884857178 proc.cpu.threads‚17& disk.nvme0n1.out‚53966.72216796875# proc.memory.rssMB‚ 2822.28515625 disk.sda1.out‚0 network.sent‚ 2.3804693e+07 disk.rbd0.out‚0 disk.sda3.out‚ 522.70703125 disk.dm-1.out‚ 522.70703125) proc.memory.percent‚17.22586154937744 disk.sda2.out‚0 disk.dm-0.in‚144902.33203125% disk.nvme0c0n1.in‚144902.33203125 disk.sda.out‚ 522.70703125 disk.sda2.in‚0( proc.memory.availableMB‚ 429.81640625 network.recv‚ 3.380721e+06# disk.nvme0n1.in‚144902.33203125‚³ 4Ä./ _wandbruntime‚12711  _runtime‚12711qFžÊÅ.Ä % _timestamp‚1.7842855189052052e+09 _runtime‚12711.030810391 _step‚1229 & lr-AdamW/pg1‚7.331386791924157e-05 & lr-AdamW/pg2‚7.331386791924157e-05 trainer/global_step‚14949DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138489899 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2260990350184448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118566902 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15989290307072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26186871 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚14949 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2272683183005696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500472647952384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138479922 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58372612 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118566904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15790550059632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257046824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2272683200045056 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15815016003584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15790890609522 _step‚1230 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26186868 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2288497857670144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257056803 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58372340 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138479922 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138489899 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2260990350184448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118566902 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15989290307072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26186871 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚14949 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2272683183005696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500472647952384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138479922 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58372612 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118566904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15790550059632 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257046824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2272683200045056 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15815016003584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15790890609522 _step‚1230 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26186868 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2288497857670144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257056803 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58372340 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138479922 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784285519517999e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15791561113354 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118574534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26188613 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257063508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2272830441074688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15816028637696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15815687291392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58376398 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58376116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257073501 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516496404065280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2288646469712384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138498965 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚516496292688896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2276928421969674 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚12711.64379252 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118574534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26188616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500506095323136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118574536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138488974 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257063508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2272830441074688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15816028637696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15815687291392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58376398 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58376116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257073501 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516496404065280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2288646469712384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138498965 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚516496292688896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2276928421969674 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚12711.64379252 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118574534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26188616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500506095323136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118574536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138488974 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784285519517999e+09Ï ÌpÏ$4Í./ _wandbruntime‚12711  _runtime‚12711”ZëÎ.‹ _step‚1232 0loss/FRt2/assignment_loss‚0.7454808960089813 trainer/global_step‚14949 % _timestamp‚1.7842855195208821e+09 0loss/FRt1/assignment_loss‚1.5166483989690533 /loss/FRt1/detection_loss‚0.5266148094575862 1loss/SRqqt2/detection_loss‚0.5815604496613445 /loss/FBt1/detection_loss‚0.4759739866183177 epoch‚37 _runtime‚12711.646223058 3loss/SRqqt1/assignment_loss‚0.38281891881723484 3loss/FBt1/assignment_loss‚0.0041665308630165895 &loss/total_loss‚7.6633599448469685 3loss/SRqqt2/assignment_loss‚0.44147821930288567 2loss/FBt2/assignment_loss‚0.036134702367904836 /loss/FRt2/detection_loss‚1.1793408493493025 1loss/SRqqt1/detection_loss‚0.8521451242041175 /loss/FBt2/detection_loss‚0.9209970592272233veTDÏ. 0loss/FRt1/assignment_loss‚1.5166483989690533 /loss/FRt1/detection_loss‚0.5266148094575862 1loss/SRqqt2/detection_loss‚0.5815604496613445 /loss/FBt1/detection_loss‚0.4759739866183177 epoch‚37 _runtime‚12711.646223058 3loss/SRqqt1/assignment_loss‚0.38281891881723484 3loss/FBt1/assignment_loss‚0.0041665308630165895 &loss/total_loss‚7.6633599448469685 3loss/SRqqt2/assignment_loss‚0.44147821930288567 2loss/FBt2/assignment_loss‚0.036134702367904836 /loss/FRt2/detection_loss‚1.1793408493493025 1loss/SRqqt1/detection_loss‚0.8521451242041175 /loss/FBt2/detection_loss‚0.9209970592272233 _step‚1232 0loss/FRt2/assignment_loss‚0.7454808960089813 trainer/global_step‚14949 % _timestamp‚1.7842855195208821e+09Ð 6]žÐ.‚(:“ ×’èÒË•Ý¢ gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚233.788' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.757600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÃn°Ñ.‚(:ø ×’èÒ®±å¤ disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0) proc.memory.percent‚17.22586154937744 disk.sda3.out‚ 522.9921875! disk.nvme0n1.in‚ 144903.046875! disk.rbd1.out‚115.88720703125 disk.sda3.in‚0# disk.dm-0.out‚53976.52001953125& disk.nvme0n1.out‚53976.52001953125 disk.drbd2165.in‚0$ memory_percent‚97.37944602966309 disk.sda.out‚ 522.9921875 disk.sda2.in‚0 network.sent‚ 2.3809712e+07) disk./.usagePercent‚36.82979506002946# disk.nvme0c0n1.in‚ 144903.046875 disk.drbd2165.out‚0' proc.memory.availableMB‚ 429.3515625 disk.dm-1.out‚ 522.9921875 disk.dm-0.in‚ 144903.046875 disk.sda1.out‚0# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17 network.recv‚ 3.384217e+06( disk.nvme0c0n1.out‚53976.52001953125 disk.sda.in‚0 disk.rbd3.in‚0 cpu‚26.02298357666482 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0$ disk./.usageGB‚548.7905616760254 disk.rbd0.out‚0ñºTµ“Ò.‚(:ˆ æ’èÒ½Ô΢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œsÓaòÓ.‚(:ç æ’èÒÃœš¥ network.sent‚ 2.3872891e+07 disk.nvme0n1.in‚ 144903.875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 523.21875 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda.out‚ 523.21875 disk.dm-1.out‚ 523.21875 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 429.3515625# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 cpu‚26.022622295749017 network.recv‚ 3.388754e+06! disk.rbd1.out‚115.94970703125 disk.rbd3.out‚0" disk.dm-0.out‚53984.0107421875 disk.sda1.in‚0' disk.nvme0c0n1.out‚53984.0107421875 disk.sda2.in‚0 disk.dm-0.in‚ 144903.875% disk.nvme0n1.out‚53984.0107421875 disk.sda1.out‚0) disk./.usagePercent‚36.82980018018015$ memory_percent‚97.37944602966309$ disk./.usageGB‚548.7906379699707 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 144903.875 proc.cpu.threads‚17˜WM+“Ô.‚(:ˆ õ’èÒ¼ƒž¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05õšœûÕ.‚(:ð õ’èÒï‘è£) disk./.usagePercent‚36.82979762010481 disk.rbd0.in‚0% disk.nvme0c0n1.in‚144904.58984375 disk.sda.in‚0$ memory_percent‚97.37839698791504 cpu‚26.0222304781749 disk.drbd2165.out‚0 disk.sda3.out‚ 523.40625 network.sent‚ 2.3877952e+07 network.recv‚ 3.392376e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 429.5234375# proc.memory.rssMB‚ 2822.28515625 disk.drbd2165.in‚0 disk.dm-0.in‚144904.58984375) proc.memory.percent‚17.22586154937744 disk.sda.out‚ 523.40625 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚115.94970703125# disk.nvme0n1.in‚144904.58984375 disk.sda1.in‚0 proc.cpu.threads‚17# disk./.usageGB‚548.790599822998 disk.rbd0.out‚0 disk.dm-1.out‚ 523.40625 disk.rbd3.in‚0& disk.nvme0c0n1.out‚53993.982421875! disk.dm-0.out‚53993.982421875$ disk.nvme0n1.out‚53993.982421875 disk.sda1.out‚0[ƒ4Ö./ _wandbruntime‚12750  _runtime‚12750$ÑyÊ×.Ä & lr-AdamW/pg1‚7.194780646294514e-05 & lr-AdamW/pg2‚7.194780646294514e-05 trainer/global_step‚14999 % _timestamp‚1.7842855588520203e+09 _runtime‚12750.977628792 _step‚1233?©±èÏØ.É _runtime‚12750.977628792 _step‚1233 & lr-AdamW/pg1‚7.194780646294514e-05 & lr-AdamW/pg2‚7.194780646294514e-05 trainer/global_step‚14999 % _timestamp‚1.7842855588520203e+09Ñ õV24Ù./ _wandbruntime‚12750  _runtime‚12750!ŸÒ¯(Ú.P F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842855588531327e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2280045251526656 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138944351 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16040524383744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2295910604653568 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2295910604653568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58564148 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138934374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257892855 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚502144085878784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2268315900817408 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2295910963032064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257882876 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15865370166272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118948502 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32289761 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32290027 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _step‚1234 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138934374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58563879 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2284157407913838 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15841507096430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118948504 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2295910963032064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16040442098176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚12750.978943925 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118948504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚14999 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚518184628874752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2280045234487296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚502144104491008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚518184527976960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15865711505408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚518184628874752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2280045234487296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚502144104491008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚518184527976960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15865711505408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2295910604653568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58564148 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138934374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257892855 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚502144085878784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2268315900817408 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2295910963032064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576Ò  ÔÈÃ4Ü./ _wandbruntime‚12751  _runtime‚12751 £'Ý.‹N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58567907 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚518219080872448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2284304929639544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16041547622912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15866724139520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257909553 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚502177533249536 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1235 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2296059216695808 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2284304572050182 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138943426 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118956136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138943426 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2280192475516928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138953417 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15866382793216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32292041 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 trainer/global_step‚14999 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257909553 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2296058858310144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58567630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26275863 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16041452761600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2280192492556288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15866382793216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138953417 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32291767 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2268462411489280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15842518150264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2280192492556288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118956134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502177514637312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚12751.596448853 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1235 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2296059216695808 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2284304572050182 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138943426 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118956136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138943426 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2280192475516928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138953417 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15866382793216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32292041 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 trainer/global_step‚14999 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257909553 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2296058858310144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58567630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26275863 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16041452761600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2280192492556288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15866382793216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138953417 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32291767 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2268462411489280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15842518150264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2280192492556288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118956134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502177514637312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚12751.596448853 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58567907 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚518219080872448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2284304929639544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16041547622912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15866724139520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257909553 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚502177533249536 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58754920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119330102 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26361371 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2303323710015488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16091788344832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119330102 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚15049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26361368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2303323351636992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚12790.877563397 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139388826 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2303323351636992 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2287407303008256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476ê¯ Ê (ë.…P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16091695573504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15916065668096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58754920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119330102 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26361371 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2303323710015488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16091788344832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119330102 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚15049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26361368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2303323351636992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚12790.877563397 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139388826 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2303323351636992 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2287407303008256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968Ö ¡,ƒ´4ì./ _wandbruntime‚12791  _runtime‚12791:¹E'í.ŠN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119337736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1239 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139407869 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15893134637172 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2287554526998528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2303471963679232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119337734 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58758393 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16092708457472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2275787945082880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15917419641344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519941778689024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15917419641344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15917078295040 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2275787928043520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784285599363476e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2303471605293568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15917078295040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139397878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119337736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚519941661021184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16092807513088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258745605 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚503848971175936 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58758668 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258745605 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _runtime‚12791.489276186 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2287554526998528 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2287554526998528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2303471963679232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119337734 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58758393 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16092708457472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2275787945082880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15917419641344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519941778689024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15917419641344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15917078295040 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2275787928043520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784285599363476e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2303471605293568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15917078295040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139397878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119337736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚519941661021184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16092807513088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258745605 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚503848971175936 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58758668 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258745605 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _runtime‚12791.489276186 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2287554526998528 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1239 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139407869 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15893134637172 E±14ï./ _wandbruntime‚12791  _runtime‚12791‰ŠxÊ ð.„ %loss/total_loss‚6.954877460532181 $ _timestamp‚1.784285599366025e+09 /loss/FRt2/assignment_loss‚0.714357767314092 1loss/SRqqt1/detection_loss‚0.8325137959130936 _runtime‚12791.491370701 0loss/FRt1/assignment_loss‚1.4150834483336348 epoch‚37 trainer/global_step‚15049 _step‚1240 2loss/SRqqt2/assignment_loss‚0.4661041654675331 1loss/FBt2/assignment_loss‚0.04243575559982103 1loss/SRqqt2/detection_loss‚0.6245149358874187 /loss/FBt1/detection_loss‚0.3343031281433807 /loss/FBt2/detection_loss‚0.5328539306436662 1loss/SRqqt1/assignment_loss‚0.332258384632208 2loss/FBt1/assignment_loss‚0.013352987997280597 0loss/FRt1/detection_loss‚0.47665884072935544 /loss/FRt2/detection_loss‚1.1704403198706972DRvñ.‰ 1loss/SRqqt1/assignment_loss‚0.332258384632208 2loss/FBt1/assignment_loss‚0.013352987997280597 0loss/FRt1/detection_loss‚0.47665884072935544 /loss/FRt2/detection_loss‚1.1704403198706972 %loss/total_loss‚6.954877460532181 $ _timestamp‚1.784285599366025e+09 /loss/FRt2/assignment_loss‚0.714357767314092 1loss/SRqqt1/detection_loss‚0.8325137959130936 _runtime‚12791.491370701 0loss/FRt1/assignment_loss‚1.4150834483336348 epoch‚37 trainer/global_step‚15049 _step‚1240 2loss/SRqqt2/assignment_loss‚0.4661041654675331 1loss/FBt2/assignment_loss‚0.04243575559982103 1loss/SRqqt2/detection_loss‚0.6245149358874187 /loss/FBt1/detection_loss‚0.3343031281433807 /loss/FBt2/detection_loss‚0.5328539306436662Ø Ùœ“ò.‚(:ˆ ¢“èÒ¢‘®¥ gpu.0.gpu‚55.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê:Ð:ó.‚(:ü ¢“èÒþ㦠disk.sda.in‚0' disk.nvme0c0n1.out‚54011.7783203125 disk.sda3.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.22586154937744 proc.cpu.threads‚17! disk.rbd1.out‚116.07470703125 cpu‚26.02122496520057 disk.dm-1.out‚ 524.16015625 disk.sda3.out‚ 524.16015625 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚54011.7783203125" disk.nvme0n1.in‚144906.1328125% disk.nvme0n1.out‚54011.7783203125 disk.drbd2165.out‚0* disk./.usagePercent‚36.829797364097274 disk.sda2.out‚0 disk.dm-0.in‚144906.1328125 disk.sda1.in‚0' proc.memory.availableMB‚ 429.2890625# proc.memory.rssMB‚ 2822.28515625 network.recv‚ 3.403534e+06$ disk./.usageGB‚548.7905960083008 disk.sda.out‚ 524.16015625 disk.sda2.in‚0$ disk.nvme0c0n1.in‚144906.1328125 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd0.out‚0 network.sent‚ 2.3964351e+07 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚97.37982749938965ñ´ê“ô.‚(:ˆ ±“èÒÐÄŽ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åXÞëùõ.‚(:î ±“èÒôÝš¥ cpu‚26.020876253849142! disk.nvme0n1.out‚ 54020.609375 disk.rbd0.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 429.34765625 network.sent‚ 2.4014372e+07 network.recv‚ 3.40812e+06 disk.sda1.out‚0 disk.sda2.in‚0" memory_percent‚97.379469871521$ disk.nvme0c0n1.in‚144906.9609375 disk.rbd3.out‚0 disk.dm-0.in‚144906.9609375# proc.memory.rssMB‚ 2822.28515625 disk.rbd0.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 54020.609375 disk.dm-1.in‚0) disk./.usagePercent‚36.82979096390892! disk.rbd1.out‚116.07470703125) proc.memory.percent‚17.22586154937744$ disk./.usageGB‚548.7905006408691 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 524.35546875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.dm-0.out‚ 54020.609375" disk.nvme0n1.in‚144906.9609375 disk.sda.out‚ 524.35546875 disk.sda3.out‚ 524.35546875i žö.‚(:’ À“èÒ´Ò²¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãIÛ ÷.‚(: À“èÒãŠç¤ disk.rbd3.in‚0 network.sent‚ 2.4019426e+07 disk.sda.in‚0# disk.nvme0n1.in‚144907.67578125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚97.37992286682129 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 524.62890625 disk.sda2.out‚0$ disk./.usageGB‚548.7905006408691 disk.dm-1.out‚ 524.62890625 disk.rbd1.in‚0.0625 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144907.67578125 cpu‚26.020484543515273 proc.cpu.threads‚17! disk.rbd1.out‚116.13720703125& disk.nvme0n1.out‚54028.58935546875 disk.sda1.out‚0 disk.drbd2165.out‚0' proc.memory.availableMB‚ 429.2734375# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 disk.dm-1.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚36.82979096390892 disk.sda3.out‚ 524.62890625 disk.dm-0.in‚144907.67578125# disk.dm-0.out‚54028.58935546875( disk.nvme0c0n1.out‚54028.58935546875 network.recv‚ 3.41128e+06#ÇÇ4ø./ _wandbruntime‚12830  _runtime‚128309MÏ3Éù.à _runtime‚12830.598625509 _step‚1241 & lr-AdamW/pg1‚6.924744528649292e-05 & lr-AdamW/pg2‚6.924744528649292e-05 trainer/global_step‚15099 $ _timestamp‚1.784285638472975e+09ßeò¨Îú.È _runtime‚12830.598625509 _step‚1241 & lr-AdamW/pg1‚6.924744528649292e-05 & lr-AdamW/pg2‚6.924744528649292e-05 trainer/global_step‚15099 $ _timestamp‚1.784285638472975e+09Ù ×Õ!B4û./ _wandbruntime‚12830  _runtime‚12830k/„(ü.€P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16142948596736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15942740070246 trainer/global_step‚15099 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15966761169920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2310736456998912 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2310736098620416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2282966950965248 _runtime‚12830.603362249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505486961731584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2298909350485604 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2310736456998912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15967102509056 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139853255 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139843278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32496713 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2310736098620416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2294769337450496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119711704 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26448618 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58945334 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58945078 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2282966968004608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚1242Þôƒ (ý.…P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139843278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32496713 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2310736098620416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2294769337450496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119711704 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26448618 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58945334 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58945078 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2282966968004608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _step‚1242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16142948596736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15942740070246 trainer/global_step‚15099 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15966761169920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2310736456998912 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2310736098620416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2282966950965248 _runtime‚12830.603362249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505486961731584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2298909350485604 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2310736456998912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15967102509056 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139853255 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976Ú ‚WÛþ4þ./ _wandbruntime‚12831  _runtime‚12831Úö‘é'ÿ.ŠN YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15967773796864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259581657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259571664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119719334 _runtime‚12831.209309019 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2310884352276992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259581657 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119719334 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26450366 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119719336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2294916595519488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚×Âk×é32498755 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15968115143168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1243 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚521664358680064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58948867 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139852330 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15943751124080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784285639083482e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚505520409102336 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2299057229800560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58949121 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2299056872211198 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505520390490112 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2294916578480128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26450363 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15968115143168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2310884710662656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16143968189952 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2294916578480128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259571664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2310884710662656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32498504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2283113478676480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2294916595519488 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521664463764992å7Wè'€/N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58949121 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2299056872211198 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505520390490112 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2294916578480128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26450363 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15968115143168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2310884710662656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16143968189952 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2294916578480128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259571664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2310884710662656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32498504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2283113478676480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2294916595519488 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521664463764992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15967773796864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 259581657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259571664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119719334 _runtime‚12831.209309019 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2310884352276992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259581657 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119719334 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26450366 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119719336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2294916595519488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32498755 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15968115143168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1243 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚521664358680064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58948867 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139852330 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15943751124080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 $ _timestamp‚1.784285639083482e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚505520409102336 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2299057229800560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888Û uOï4// _wandbruntime‚12831  _runtime‚12831Wym‚/Š /loss/FBt2/detection_loss‚0.3173290037084371 trainer/global_step‚15099 2loss/SRqqt1/assignment_loss‚0.3549694979864633 4loss/FBt2/assignment_loss‚0.00019306048917102207 /loss/FRt1/detection_loss‚0.5044307139855139 &loss/total_loss‚7.0243832585669015 % _timestamp‚1.7842856390873451e+09 _runtime‚12831.212675373 1loss/SRqqt1/detection_loss‚0.8139913984417264 1loss/SRqqt2/detection_loss‚0.6122719995362987 _step‚1244 epoch‚37 /loss/FRt1/assignment_loss‚1.524751871856207 0loss/FRt2/assignment_loss‚0.7500817777917665 2loss/FBt1/assignment_loss‚0.004442502555258937 /loss/FBt1/detection_loss‚0.4539272192243278 3loss/SRqqt2/assignment_loss‚0.45200526115367146 /loss/FRt2/detection_loss‚1.23598895183806021 zŒƒ/ 1loss/SRqqt1/detection_loss‚0.8139913984417264 1loss/SRqqt2/detection_loss‚0.6122719995362987 _step‚1244 epoch‚37 /loss/FRt1/assignment_loss‚1.524751871856207 0loss/FRt2/assignment_loss‚0.7500817777917665 2loss/FBt1/assignment_loss‚0.004442502555258937 /loss/FBt1/detection_loss‚0.4539272192243278 3loss/SRqqt2/assignment_loss‚0.45200526115367146 /loss/FRt2/detection_loss‚1.2359889518380602 /loss/FBt2/detection_loss‚0.3173290037084371 trainer/global_step‚15099 2loss/SRqqt1/assignment_loss‚0.3549694979864633 4loss/FBt2/assignment_loss‚0.00019306048917102207 /loss/FRt1/detection_loss‚0.5044307139855139 &loss/total_loss‚7.0243832585669015 % _timestamp‚1.7842856390873451e+09 _runtime‚12831.212675373Ü "W7“„/‚(:ˆ Ï“èÒåšž£ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚084ª=…/‚(:ö Ï“èҷ¤¥$ disk./.usageGB‚548.7904434204102 disk.sda1.in‚0 disk.sda3.out‚ 525.24609375 network.sent‚ 2.4024428e+07 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0( disk./.usagePercent‚36.8297871237959 disk.sda.out‚ 525.24609375 disk.dm-1.in‚0! disk.rbd1.out‚116.13720703125 disk.rbd3.out‚0& proc.memory.availableMB‚ 429.078125# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744% disk.nvme0c0n1.out‚54034.48046875 disk.dm-1.out‚ 525.24609375 cpu‚26.020148213314105# disk.nvme0n1.in‚144907.67578125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-0.in‚144907.67578125 network.recv‚ 3.414889e+06 disk.drbd2165.in‚0# memory_percent‚97.3811149597168# disk.nvme0n1.out‚54034.48046875% disk.nvme0c0n1.in‚144907.67578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚54034.48046875 proc.cpu.threads‚17Þ÷P†/‚(:’ Þ“èÒâÔ¸¢ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?‹¿ðõ‡/‚(:ê Þ“èÒÍöé¤ disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0# disk.nvme0n1.in‚144908.50390625 disk.drbd2165.in‚0 disk.dm-0.in‚144908.50390625) disk./.usagePercent‚36.82975333080138 disk.rbd0.in‚0 disk.sda.out‚525.5$ memory_percent‚97.37982749938965 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144908.50390625' disk.nvme0c0n1.out‚54041.2314453125 disk.dm-1.out‚525.5! disk.rbd1.out‚116.19970703125 disk.rbd3.out‚0 cpu‚26.019797295200206$ disk./.usageGB‚548.7899398803711 disk.rbd1.in‚0.0625 network.sent‚ 2.4087609e+07 network.recv‚ 3.419252e+06 disk.rbd3.in‚0' proc.memory.availableMB‚ 429.2890625) proc.memory.percent‚17.22586154937744% disk.nvme0n1.out‚54041.2314453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0" disk.dm-0.out‚54041.2314453125# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚525.5 disk.dm-1.in‚0ÐÁ•¥ˆ/‚(:’ í“èÒ¾¤ª¢ gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚242.43' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.486000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ájÿ‰/‚(:ô í“èÒØܤ disk.dm-0.in‚ 144909.21875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 525.72265625 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2822.28515625( disk./.usagePercent‚36.8298147726096 disk.rbd3.in‚0" disk.dm-0.out‚54050.3818359375 network.sent‚ 2.4092616e+07% disk.nvme0n1.out‚54050.3818359375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 525.72265625 disk.sda2.out‚0 disk.sda3.out‚ 525.72265625! disk.rbd1.out‚116.19970703125 network.recv‚ 3.422675e+06" disk.nvme0c0n1.in‚ 144909.21875 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚548.7908554077148 disk.rbd0.out‚0' disk.nvme0c0n1.out‚54050.3818359375 disk.sda3.in‚0 disk.nvme0n1.in‚ 144909.21875$ memory_percent‚97.38192558288574' proc.memory.availableMB‚ 428.9453125) proc.memory.percent‚17.22586154937744 cpu‚26.01946504177724 disk.rbd3.out‚0Pl/]4Š// _wandbruntime‚12870  _runtime‚12870Hªë×Ê‹/Ä _step‚1245 & lr-AdamW/pg1‚6.791331900548929e-05 & lr-AdamW/pg2‚6.791331900548929e-05 trainer/global_step‚15149 % _timestamp‚1.7842856783592029e+09 _runtime‚12870.485013171Z?ךÏŒ/É & lr-AdamW/pg1‚6.791331900548929e-05 & lr-AdamW/pg2‚6.791331900548929e-05 trainer/global_step‚15149 % _timestamp‚1.7842856783592029e+09 _runtime‚12870.485013171 _step‚1245Ý j¼F4// _wandbruntime‚12870  _runtime‚12870ÅT©Fv Ž/P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260391032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26535868 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚15149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16017456671744 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15993356557154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15993016007264 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _step‚1246 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140297730 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120093302 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2318149203982336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507158399657984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140297730 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2302131388932096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140307707 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32600459 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16194087545856 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2306285500566112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59136327 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16194182414336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2290292501598208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16017798010880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2306285858155362 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2290292484558848 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2318148845603840 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26535868 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚15149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16017456671744 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15993356557154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15993016007264 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _step‚1246 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140297730 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120093302 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2318149203982336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507158399657984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140297730 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2302131388932096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140307707 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32600459 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16194087545856 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2306285500566112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59136327 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16194182414336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2290292501598208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16017798010880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2306285858155362 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2290292484558848 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2318148845603840 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16195108070400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120100936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59140386 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32602770 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚523386936486912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260407716 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140306782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26537613 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507191847028736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26537616 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260417709 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2302278647001088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2318297457646080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16018810644992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59140112 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2302278629961728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2306433022291706 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2302278647001088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2318297457646080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120100934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842856789723663e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260417709 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120100936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32602499 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16018469298688 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140316773 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15994367610988 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚12871.09816609 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260407716 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59140112 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2302278629961728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2306433022291706 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2302278647001088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2318297457646080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120100934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842856789723663e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260417709 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120100936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32602499 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16018469298688 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140316773 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15994367610988 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚12871.09816609 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260407716 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16195108070400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120100936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59140386 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32602770 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚523386936486912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260407716 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140306782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26537613 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507191847028736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26537616 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260417709 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2302278647001088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2318297457646080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591ß í®z4“// _wandbruntime‚12871  _runtime‚12871½¾Á¥”/ˆ _runtime‚12871.10118513 /loss/FRt2/detection_loss‚1.1942536096303427 epoch‚37 trainer/global_step‚15149 # _timestamp‚1.78428567897586e+09 _step‚1248 /loss/FRt2/assignment_loss‚0.768828648112037 /loss/FBt1/detection_loss‚0.4432392249070283 3loss/SRqqt1/assignment_loss‚0.43035567589609675 2loss/FBt1/assignment_loss‚0.017338976696135898 4loss/FBt2/assignment_loss‚0.00Q0hÓi013807916844173334 1loss/SRqqt2/detection_loss‚0.5197146040160531 0loss/FBt2/detection_loss‚0.46651477650315926 0loss/FRt1/assignment_loss‚1.5561363880549617 3loss/SRqqt2/assignment_loss‚0.42330494670070345 /loss/FRt1/detection_loss‚0.4931829457798145 1loss/SRqqt1/detection_loss‚0.8496738772504032 %loss/total_loss‚7.162681752715178§ø•/ 3loss/SRqqt2/assignment_loss‚0.42330494670070345 /loss/FRt1/detection_loss‚0.4931829457798145 1loss/SRqqt1/detection_loss‚0.8496738772504032 %loss/total_loss‚7.162681752715178 _runtime‚12871.10118513 /loss/FRt2/detection_loss‚1.1942536096303427 epoch‚37 trainer/global_step‚15149 # _timestamp‚1.78428567897586e+09 _step‚1248 /loss/FRt2/assignment_loss‚0.768828648112037 /loss/FBt1/detection_loss‚0.4432392249070283 3loss/SRqqt1/assignment_loss‚0.43035567589609675 2loss/FBt1/assignment_loss‚0.017338976696135898 4loss/FBt2/assignment_loss‚0.00013807916844173334 1loss/SRqqt2/detection_loss‚0.5197146040160531 0loss/FBt2/detection_loss‚0.46651477650315926 0loss/FRt1/assignment_loss‚1.5561363880549617à ö(È‘–/‚(:† ü“èÒãí­£ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í¢ÞNü—/‚(:ñ ü“èÒõúž¥$ disk./.usageGB‚548.7907295227051 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚36.82980632436097 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 cpu‚26.019111585257313 disk.rbd0.out‚0' disk.nvme0c0n1.out‚54057.8935546875 disk.sda2.out‚0 disk.dm-0.in‚ 144910.046875 proc.cpu.threads‚17 network.recv‚ 3.425853e+06 disk.sda.out‚ 525.96875 disk.dm-1.out‚ 525.96875 disk.rbd3.in‚0( proc.memory.availableMB‚ 428.95703125) proc.memory.percent‚17.22586154937744 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0! disk.rbd1.out‚116.32470703125$ memory_percent‚97.38185405731201# proc.memory.rssMB‚ 2822.28515625# disk.nvme0c0n1.in‚ 144910.046875! disk.nvme0n1.in‚ 144910.046875% disk.nvme0n1.out‚54057.8935546875 network.sent‚ 2.4097665e+07 disk.rbd0.in‚0 disk.sda3.out‚ 525.96875" disk.dm-0.out‚54057.8935546875 disk.sda1.out‚0²²š¸‘˜/‚(:† ‹”èÒבÁ¢ gpu.0.gpu‚50.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÌç™/‚(:ý ‹”èÒÙÆó£ disk.rbd1.in‚0.0625 disk.sda3.in‚0 disk.dm-1.out‚ 526.40234375 disk.dm-0.in‚ 144910.046875$ memory_percent‚97.38190174102783( proc.memory.availableMB‚ 428.94921875 cpu‚26.018766779602345 disk.rbd3.out‚0! disk.nvme0n1.in‚ 144910.046875 disk.drbd2165.out‚0 network.sent‚ 2.4160736e+07$ disk./.usageGB‚548.7870178222656# disk.dm-0.out‚54063.23486328125& disk.nvme0n1.out‚54063.23486328125# disk.nvme0c0n1.in‚ 144910.046875! disk.rbd1.out‚116.32470703125 disk.sda.in‚0 disk.sda3.out‚ 526.40234375 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2822.28515625 network.recv‚ 3.430678e+06 disk.sda.out‚ 526.40234375 disk.sda2.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚36.82955722903016 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚54063.23486328125) proc.memory.percent‚17.22586154937744 proc.cpu.threads‚17NÁö4š// _wandbruntime‚12910  _runtime‚12910›W€Ê›/Ä trainer/global_step‚15199 % _timestamp‚1.7842857183172772e+09 _runtime‚12910.442821724 _step‚1249 & lr-AdamW/pg1‚6.659001007697629e-05 & lr-AdamW/pg2‚6.659001007697629e-05àjÏœ/É & lr-AdamW/pg2‚6.659001007697629e-05 trainer/global_step‚15199 % _timestamp‚1.7842857183172772e+09 _runtime‚12910.442821724 _step‚1249 & lr-AdamW/pg1‚6.659001007697629e-05á tÀêí4// _wandbruntime‚12910  _runtime‚12910±u¯«(ž/P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26623118 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 261227084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 261237063 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16068493512704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140752182 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508829856196608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2325561592587264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2297618035191808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2313662008235870 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2309493440413696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16068152173568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 % _timestamp‚1.7842857183182592e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32704570 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16245292792832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2325561950965760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140752182 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2309493440413696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚508829837584384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59327429 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2325561592587264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚525075233372160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚525075130377216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2309493457453056 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59327691 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2313661650646620 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888J;ª´ (Ÿ/†P C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 261227084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 261237063 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16068493512704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140752182 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508829856196608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2325561592587264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2297618035191808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2313662008235870 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2309493440413696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16068152173568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 % _timestamp‚1.7842857183182592e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32704570 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16245292792832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2325561950965760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140752182 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2309493440413696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚508829837584384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59327429 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2325561592587264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚525075233372160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚525075130377216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2309493457453056 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59327691 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2313661650646620 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26623118 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136â ñÃNˆ4 // _wandbruntime‚12911  _runtime‚12911¡ˆ¨x´¡/‹N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261243768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140771225 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508863266342912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16069164800512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120482536 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2309640698482688 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16069506146816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2325710204629504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140771225 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120482534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2309640681443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59331223 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 261243768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16044984097896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2325710204629504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16246398494208 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2313809172372214 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26624863 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 261253761 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2325709846243840 REDeviceStatsMonitor.on_train_batch_end/inactive]Ä3I] _split.small_pool.freed‚32706360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525109683449344 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2313809529961576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚15199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26624866 _runtime‚12911.055350636 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120482536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚1251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16069506146816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16069164800512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261253761 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32706626 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120482534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2309640681443328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59331223 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 261243768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16044984097896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2325710204629504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16246398494208 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2313809172372214 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26624863 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 261253761 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2325709846243840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32706360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525109683449344 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2313809529961576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚15199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26624866 _runtime‚12911.055350636 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120482536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚1251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16069506146816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16069164800512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261253761 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32706626 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508863266342912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16069164800512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120482536 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2309640698482688 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16069506146816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2325710204629504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140771225 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 Q&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6268329620361328 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6687399744987488 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975671172142029 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897577166557312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8985124230384827 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24963881075382233 )particle/sensitivity‚0.61677086353302 epoch‚37 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26654577255249023 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5857873558998108 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6511679887771606 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.682445228099823 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3404940068721771 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6195434331893921 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046594612300395966 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28440648317337036 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 5validation_average_jet_accuracy‚0.380718469619751 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6343050599098206 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,jet/accuracy_3_of_3‚0.058797985315322876 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.017038017511367798 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7810152769088745 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 )jet/accuracy_2_of_4‚0.797858476638794 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9075230360031128 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.657775342464447 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.01799626648426056 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6694693565368652 particle/accuracy_1_of_4‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6184257864952087 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6782501935958862 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.759963870048523 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33081069588661194 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2157983034849167 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.164166659116745 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2597155272960663 $ _timestamp‚1.784285736045314e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24779489636421204 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27472853660583496 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6567732691764832 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6368781328201294 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23674580454826355 _step‚1253 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03807900473475456 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04360954836010933 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.699560821056366 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.016880661249160767 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8473066687583923 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3381996154785156 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5669264197349548 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'particle/f_score‚0.6270530819892883 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25845327973365784 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6403470039367676 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9918553233146667 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3671373724937439 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16140970587730408 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875767827033997 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29427608847618103 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08171762526035309 particle/accuracy_1_of_2‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.71875 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2856655716896057 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20754645764827728 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018432680517435074 *jet/accuracy_1_of_4‚0.9804345369338989 +particle/specificity‚0.9041789174079895 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38150060176849365 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23186099529266357 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6116272211074829 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_5_of_5‚1 _runtime‚12928.171561792 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7256585359573364 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5581910014152527 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2106683850288391 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.8015034794807434 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23347477614879608 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6192932724952698 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9283708930015564 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6125171184539795 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4910121262073517 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6406230926513672 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2662411034107208 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6233935952186584 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.847197949886322 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4585520327091217 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7021467089653015 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 *jet/accuracy_1_of_2‚0.7980690002441406 *jet/accuracy_2_of_3‚0.5850765109062195 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 jet/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49097269773483276 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6177903413772583 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3071461021900177 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6704550981521606 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3673231303691864 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946258068084717 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7340822219848633 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 particle/accuracy_3_of_3‚1 trainer/global_step‚15199 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24794621765613556 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4983901083469391 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.53083336353302 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7253968119621277 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21890099346637726 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.807270348072052 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24311599135398865 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05161071568727493 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.275360107421875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018432680517435074 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.865484893321991®M¢ª/ùz 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29427608847618103 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08171762526035309 particle/accuracy_1_of_2‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.71875 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2856655716896057 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20754645764827728 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.018432680517435074 *jet/accuracy_1_of_4‚0.9804345369338989 +particle/specificity‚0.9041789174079895 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38150060176849365 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23186099529266357 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6116272211074829 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_5_of_5‚1 _runtime‚12928.171561792 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7256585359573364 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5581910014152527 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2106683850288391 9"Purity/1FRt2S‚!±)â7Rqqt*FBt/SRqqt_purity‚0.8015034794807434 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23347477614879608 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6192932724952698 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9283708930015564 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6125171184539795 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4910121262073517 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6406230926513672 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2662411034107208 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6233935952186584 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.847197949886322 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4585520327091217 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7021467089653015 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 *jet/accuracy_1_of_2‚0.7980690002441406 *jet/accuracy_2_of_3‚0.5850765109062195 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 jet/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49097269773483276 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6177903413772583 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3071461021900177 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6704550981521606 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3673231303691864 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946258068084717 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7340822219848633 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 particle/accuracy_3_of_3‚1 trainer/global_step‚15199 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24794621765613556 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4983901083469391 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.53083336353302 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7253968119621277 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21890099346637726 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.807270348072052 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24311599135398865 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05161071568727493 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.275360107421875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.018432680517435074 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.865484893321991 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970018267631531 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27728405594825745 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6718931198120117 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28420406579971313 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27814731001853943 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6394877433776855 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5546666979789734 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6965987086296082 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6782501935958862 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5386840105056763 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2237856686115265 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24750497937202454 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09742973744869232 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.865484893321991 )loss/val_total_loss‚8.049032592522616 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08085562288761139 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7742360234260559 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7596161961555481 /particle/accuracy_4_of_4‚0.9493521451950073 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6922902464866638 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.017868107184767723 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6062413454055786 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27472853660583496 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6623173952102661 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 (particle/accuracy‚0.8424595594406128 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6687337160110474 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953467845916748 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6121543049812317 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993925929069519 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6195434331893921 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6268329620361328 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6687399744987488 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975671172142029 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897577166557312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8985124230384827 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24963881075382233 )particle/sensitivity‚0.61677086353302 epoch‚37 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26654577255249023 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5857873558998108 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6511679887771606 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.682445228099823 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3404940068721771 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6195434331893921 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046594612300395966 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28440648317337036 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 5validation_average_jet_accuracy‚0.380718469619751 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6343050599098206 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ,jet/accuracy_3_of_3‚0.058797985315322876 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.017038017511367798 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7810152769088745 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 )jet/accuracy_2_of_4‚0.797858476638794 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9075230360031128 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.657775342464447 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.01799626648426056 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6694693565368652 particle/accuracy_1_of_4‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6184257864952087 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6782501935958862 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.759963870048523 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33081069588661194 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2157983034849167 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.164166659116745 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2597155272960663 $ _timestamp‚1.784285736045314e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24779489636421204 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27472853660583496 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6567732691764832 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6368781328201294 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23674580454826355 _step‚1253 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03807900473475456 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04360954836010933 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.699560821056366 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.016880661249160767 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8473066687583923 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3381996154785156 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5669264197349548 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'particle/f_score‚0.6270530819892883 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25845327973365784 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6403470039367676 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9918553233146667 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3671373724937439 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16140970587730408 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875767827033997å …;YÆž«/‚(:“ ©”èÒ­Èæ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.293599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ÐR¬/‚(:ø ©”èÒú¤å¤ disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.sda1.out‚0) proc.memory.percent‚17.22586154937744) disk./.usagePercent‚36.83009484485206 disk.dm-1.out‚ 527.109375 disk.rbd0.out‚0 cpu‚26.023324473896952% disk.nvme0c0n1.in‚144911.58984375' disk.nvme0c0n1.out‚54082.0244140625 disk.sda.in‚0 disk.dm-0.in‚144911.58984375 disk.rbd0.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚548.7950286865234 disk.sda.out‚ 527.109375# proc.memory.rssMB‚ 2822.28515625 network.sent‚ 2.4227982e+07 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚144911.58984375% disk.nvme0n1.out‚54082.0244140625& proc.memory.availableMB‚ 428.515625 network.recv‚ 3.437924e+06 disk.rbd3.in‚0" disk.dm-0.out‚54082.0244140625 disk.drbd2165.in‚0$ memory_percent‚97.38454818725586 disk.sda3.out‚ 527.109375! disk.rbd1.out‚116.44970703125 disk.sda1.in‚0 disk.drbd2165.out‚0ÁdA“­/‚(:ˆ ¸”èÒüüÔ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚268.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05Óü®/‚(:õ ¸”èÒÇ÷«¥ disk.rbd1.in‚0.0625 disk.dm-0.in‚144912.3046875 network.recv‚ 3.442242e+06) disk./.usagePercent‚36.83009382082192 cpu‚26.0229295789082& disk.nvme0n1.out‚54090.23876953125 disk.sda.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚116.51220703125$ memory_percent‚97.38311767578125# proc.memory.rssMB‚ 2822.28515625# disk.dm-0.out‚54090.23876953125 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚144912.3046875( disk.nvme0c0n1.out‚54090.23876953125" proc.memory.availableMB‚428.75$ disk./.usageGB‚548.7950134277344 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 527.5390625 proc.cpu.threads‚17" disk.nvme0n1.in‚144912.3046875 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda.out‚ 527.5390625) proc.memory.percent‚17.22586154937744 network.sent‚ 2.4285315e+07 disk.sda2.in‚0 disk.sda3.out‚ 527.5390625 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0FÏAQ“¯/‚(:ˆ Ç”èÒäñ¢ gpu.0.gpu‚90.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚256.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.2796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{t<¤°/‚(:÷ Ç”èÒ’¿¤* disk./.usagePercent‚36.829668080292485 disk.dm-0.in‚144913.1328125% disk.nvme0n1.out‚54101.9833984375$ disk.nvme0c0n1.in‚144913.1328125$ disk./.usageGB‚548.7886695861816" disk.dm-0.out‚54101.9833984375 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚97.38471508026123 disk.sda2.out‚0 network.sent‚ 2.4290586e+07! disk.rbd1.out‚116.51220703125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2822.28515625 disk.dm-1.out‚ 528.046875" disk.nvme0n1.in‚144913.1328125 disk.sda1.in‚0) proc.memory.percent‚17.22586154937744 disk.rbd1.in‚0.0625' disk.nvme0c0n1.out‚54101.9833984375 disk.sda.out‚ 528.046875 network.recv‚ 3.446001e+06 disk.dm-1.in‚0 disk.sda3.out‚ 528.046875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚26.02259916860287 disk.rbd3.in‚0( proc.memory.availableMB‚ 428.48828125í9Bb4±// _wandbruntime‚12967  _runtime‚12967²¿6ɲ/à & lr-AdamW/pg1‚6.527760349461366e-05 & lr-AdamW/pg2‚6.527760349461366e-05 trainer/global_step‚15249 $ _timestamp‚1.784285775420023e+09 _runtime‚12967.545603249 _step‚1254ќ҄γ/È & lr-AdamW/pg2‚6.527760349461366e-05 trainer/global_step‚15249 $ _timestamp‚1.784285775420023e+09 _runtime‚12967.545603249 _step‚1254 & lr-AdamW/pg1‚6.527760349461366e-05æ °e64´// _wandbruntime‚12967  _runtime‚12967ŽK/Ï(µ/ÿO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141286933 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16136922220032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2318785058545664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59552424 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2318785075585024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2306852214960128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2322964541255687 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2334921639426560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16314701742592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2334921997805056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16136580880896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141286933 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16314613165568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141276956 _runtime‚12967.550361962 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16136580880896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262254124 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120977170 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510517206222848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2318785075585024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16111968706309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59552155 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚15249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2334921997805056 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2322964183666437 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2306852231999488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120977168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510517187610624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16136922220032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32839267 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262264103 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 # _timestamp‚1.78428577542103e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120977168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26712888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚15249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2334921997805056 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2322964183666437 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2306852231999488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120977168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510517187610624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16136922220032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32839267 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262264103 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 # _timestamp‚1.78428577542103e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120977168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26712888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2334921639426560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16314701742592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2334921997805056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16136580880896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141286933 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16314613165568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141276956 _runtime‚12967.550361962 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16136580880896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262254124 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120977170 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510517206222848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2318785075585024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16111968706309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59552155 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16315718563840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2323111705392031 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120984800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26714633 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120984802 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16112979760031 % _timestamp‚1.7842857760326867e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32841327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510550616369152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16113320310033 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120984802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32841599 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16315623702528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16137593507840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262280801 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141286008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59555960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262270808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141286008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141295999 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16137934854144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2318932316614656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510550634981376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2318932299575296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16137934854144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141295999 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526866240071680 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2306998742671360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚15249 _runtime‚12968.158522069 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2306998725632000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59556235 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872‘źÇ'¹/N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16137593507840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59555960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262270808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141286008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141295999 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16137934854144 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2318932316614656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510550634981376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2318932299575296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16137934854144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141295999 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526866240071680 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2306998742671360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚15249 _runtime‚12968.158522069 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2306998725632000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59556235 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16315718563840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2323111705392031 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120984800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26714633 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120984802 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16112979760031 % _timestamp‚1.7842857760326867e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32841327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510550616369152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16113320310033 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120984802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32841599 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16315623702528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16137593507840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262280801 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141286008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EîE$å¾/‚(:Ú Ö”èÒ¹…¥! disk.nvme0n1.out‚ 54105.515625 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 54105.515625# proc.memory.rssMB‚ 2822.28515625" disk.nvme0n1.in‚144913.1328125! disk.rbd1.out‚116.57470703125 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚144913.1328125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-0.in‚144913.1328125 network.recv‚ 3.449248e+06# disk./.usageGB‚548.788703918457 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 cpu‚26.02226360908052 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.out‚528.5) proc.memory.percent‚17.22586154937744 disk.dm-0.out‚ 54105.515625 disk.rbd1.in‚0.0625 disk.sda2.out‚0$ memory_percent‚97.38314151763916) disk./.usagePercent‚36.82967038436029 disk.rbd3.out‚0( proc.memory.availableMB‚ 428.74609375 disk.sda.out‚528.5 disk.sda2.in‚0 disk.sda3.in‚0 network.sent‚ 2.4295486e+07 disk.dm-1.out‚528.5XÉ78‘¿/‚(:† å”èÒÕó¨¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚235.703' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.1406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z'É À/‚(:ÿ å”èÒ¢ù­¥ disk.sda3.out‚ 528.81640625' proc.memory.availableMB‚ 428.6171875 network.sent‚ 2.4360811e+07 disk.rbd3.out‚0# disk.nvme0n1.in‚144913.84765625 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚54117.2607421875 disk.sda1.in‚0 cpu‚26.021910686944057$ disk./.usageGB‚548.7994041442871 disk.rbd3.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144913.84765625 disk.dm-1.out‚ 528.81640625* disk./.usagePercent‚36.830388485493835' disk.nvme0c0n1.out‚54117.2607421875 disk.sda2.out‚0 disk.dm-1.in‚0 network.recv‚ 3.454234e+06 disk.rbd0.in‚0 disk.sda.out‚ 528.81640625 disk.rbd1.in‚0.0625! disk.rbd1.out‚116.57470703125% disk.nvme0n1.out‚54117.2607421875# memory_percent‚97.3839282989502 disk.dm-0.in‚144913.84765625 disk.sda1.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 disk.drbd2165.out‚0]ѨžÁ/‚(:“ ô”èÒÀ«û£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Ú%Â/‚(:ú ô”èÒ‘¤Í¤ disk.sda.in‚0 disk.sda2.out‚0# disk.nvme0n1.in‚144914.67578125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0.0625 disk.dm-0.out‚54124.03515625 proc.cpu.threads‚17# disk.nvme0n1.out‚54124.03515625 disk.drbd2165.in‚0$ memory_percent‚97.38385677337646# proc.memory.rssMB‚ 2822.28515625 network.sent‚ 2.4363775e+07 disk.sda1.out‚0% disk.nvme0c0n1.out‚54124.03515625 disk.rbd3.in‚0) proc.memory.percent‚17.22586154937744 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 529.19140625 disk.rbd3.out‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144914.67578125 disk.sda2.in‚0 disk.sda3.out‚ 529.19140625 disk.dm-0.in‚144914.67578125* disk./.usagePercent‚36.830430726736985/–Ûà disk.rbd0.in‚0! disk.rbd1.out‚116.63720703125 network.recv‚ 3.457481e+06$ disk./.usageGB‚548.8000335693359 disk.sda.out‚ 529.19140625( proc.memory.availableMB‚ 428.62890625 cpu‚26.02152461615427R°«Í4Ã// _wandbruntime‚13007  _runtime‚13007œ¶‚yÉÄ/à _step‚1258 & lr-AdamW/pg1‚6.397618355182363e-05 & lr-AdamW/pg2‚6.397618355182363e-05 trainer/global_step‚15299 $ _timestamp‚1.784285815200907e+09 _runtime‚13007.326522144Y«±bÎÅ/È & lr-AdamW/pg1‚6.397618355182363e-05 & lr-AdamW/pg2‚6.397618355182363e-05 trainer/global_step‚15299 $ _timestamp‚1.784285815200907e+09 _runtime‚13007.326522144 _step‚1258ê Œ™4Æ// _wandbruntime‚13007  _runtime‚130078'g (Ç/€P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16187617721856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121358768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32943674 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16187617721856 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2314177748553728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121358770 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263100155 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2342334744788480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121358770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 $ _timestamp‚1.784285815201962e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2326147110027264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16187276382720 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141741385 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2342334386409984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121358768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1259 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16365810713600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141731408 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2326147110027264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141731408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16162925743107 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2326147127066624 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2326147127066624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚13007.327747507 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141741385 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2330340333746945 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263090176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59743812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2314177765593088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚512188644149248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26800141 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2330340691336195 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚15299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2342334386409984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528554539245568 B§ (È/…P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2342334744788480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121358770 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 $ _timestamp‚1.784285815201962e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2326147110027264 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16187276382720 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141741385 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2342334386409984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121358768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1259 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16365810713600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141731408 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2326147110027264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141731408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16162925743107 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2326147127066624 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2326147127066624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚13007.327747507 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141741385 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2330340333746945 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263090176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59743812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2314177765593088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚512188644149248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26800141 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2330340691336195 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚15299 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2342334386409984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528554539245568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121358768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32943674 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16187617721856 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2314177748553728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121358770 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263100155 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438ë èÒåe4É// _wandbruntime‚13007  _runtime‚13007‹¯Nç'Ê/‰N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16163596246939 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1260 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2326294368096256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2330488213061901 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263106860 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263116853 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59747668 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2330487855472539 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141750451 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26801883 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 # _timestamp‚1.78428581581264e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16188289009664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528588877430272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32946046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528588988806656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2326294368096256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512222072907776 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16188289009664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59747932 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16163936796941 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26801886 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121366402 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263116853 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121366402 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2342482998452224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16366823134720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2326294351056896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121366400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121366400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16366915898880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚15299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263106860 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141740460 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2330488213061901 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263106860 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263116853 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59747668 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2330487855472539 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141750451 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26801883 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 # _timestamp‚1.78428581581264e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16188289009664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528588877430272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32946046 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528588988806656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2326294368096256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512222072907776 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16188289009664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59747932 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16163936796941 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26801886 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121366402 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263116853 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121366402 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2342482998452224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16366823134720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2326294351056896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121366400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121366400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16366915898880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚15299 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263106860 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141740460 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842858552652864e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142185860 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16213201680125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142195837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530277178258432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33048676 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121740368 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2337716483827453 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121740370 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2333509161508864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2349747133393408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2321503299186688 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16238313223680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2333509161508864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16238313223680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263926228 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121740370 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26887388 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513860063463424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530277064777728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250tMx"Ø/†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7842858552652864e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142185860 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16213201680125 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142195837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530277178258432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33048676 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121740368 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2337716483827453 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121740370 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2333509161508864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2349747133393408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2321503299186688 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16238313223680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2333509161508864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16238313223680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263926228 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121740370 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26887388 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513860063463424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530277064777728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121740368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263936207 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33048948 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _runtime‚13047.391069715 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_spléÃ6C”it.all.freed‚59936064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2349747133393408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2349747491771904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2333509178548224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16237971884544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2337864005553047 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121748002 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2349895387049984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142204903 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚13048.006287495 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26889136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16214212733847 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121748000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2333656402538496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263952905 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121748000 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2349895745435648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33050721 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2349895745435648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530311623180800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263952905 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142204903 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16239325857792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59940126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2321649809858560 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16214553283849 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16239325857792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121748002 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142194912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16418015388160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59939854 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2333656419577856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26889133 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269SYA˜'Û/N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59939854 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2333656419577856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26889133 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2337864005553047 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121748002 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2349895387049984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142204903 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚13048.006287495 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26889136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16214212733847 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121748000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2333656402538496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263952905 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121748000 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2349895745435648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33050721 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2349895745435648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530311623180800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263952905 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142204903 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16239325857792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59940126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2321649809858560 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16214553283849 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16239325857792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121748002 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142194912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16418015388160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664ð õ&QÀ4Ü// _wandbruntime‚13048  _runtime‚13048òV Ý/… -loss/FRt2/assignment_loss‚0.8450089137939 2loss/SRqqt1/assignment_loss‚0.5082339360248731 /loss/FRt1/detection_loss‚0.4933291638358835 0loss/SRqqt2/detection_loss‚0.654069714567727 _step‚1265 3loss/SRqqt2/assignment_loss‚0.38065542264215135 1loss/SRqqt1/detection_loss‚0.8729606521102364 /loss/FBt1/detection_loss‚0.3574665519197301 %loss/total_loss‚7.334215580508726 epoch‚38 trainer/global_step‚15349 /loss/FRt1/assignment_loss‚1.487801455756993 2loss/FBt1/assignment_loss‚0.002226829559969914 /loss/FBt2/detection_loss‚0.5503075391894433 4loss/FBt2/assignment_loss‚0.00021502972702682526 /loss/FRt2/detection_loss‚1.1819403713807923 % _timestamp‚1.7842858558826892e+09 _runtime‚13048.008019124±V†Þ/Š /loss/FBt1/detection_loss‚0.3574665519197301 %loss/total_loss‚7.334215580508726 epoch‚38 trainer/global_step‚15349 /loss/FRt1/assignment_loss‚1.487801455756993 2loss/FBt1/assignment_loss‚0.002226829559969914 /loss/FBt2/detection_loss‚0.5503075391894433 4loss/FBt2/assignment_loss‚0.00021502972702682526 /loss/FRt2/detection_loss‚1.1819403713807923 % _timestamp‚1.7842858558826892e+09 _runtime‚13048.008019124 -loss/FRt2/assignment_loss‚0.8450089137939 2loss/SRqqt1/assignment_loss‚0.5082339360248731 /loss/FRt1/detection_loss‚0.4933291638358835 0loss/SRqqt2/detection_loss‚0.654069714567727 _step‚1265 3loss/SRqqt2/assignment_loss‚0.38065542264215135 1loss/SRqqt1/detection_loss‚0.8729606521102364ñ ¡bƒ “ß/‚(:ˆ ¡•èÒƒÁ„¤ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï¾Jëúà/‚(:ï ¡•èÒ­ùý¥# proc.memory.rssMB‚ 2822.28515625$ memory_percent‚97.38550186157227 disk.nvme0n1.in‚ 144916.21875% disk.nvme0n1.out‚54146.8251953125 disk.sda1.in‚0 disk.drbd2165.in‚0 cpu‚26.020490416558147 disk.sda3.in‚0 disk.dm-0.in‚ 144916.21875$ disk./.usageGB‚548.7999305725098 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 529.890625! disk.rbd1.out‚116.69970703125 proc.cpu.threads‚17) disk./.usagePercent‚36.83042381453356 disk.sda.out‚ 529.890625 disk.sda3.out‚ 529.890625" disk.dm-0.out‚54146.8251953125) proc.memory.percent‚17.22586154937744 network.sent‚ 2.4436982e+07 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 144916.21875 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 network.recv‚ 3.469184e+06& proc.memory.availableMB‚ 428.359375 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚54146.8251953125 disk.sda2.out‚0 disk.rbd3.in‚0ÓÉj³“á/‚(:ˆ °•èÒ’ì÷¤ gpu.0.gpu‚62.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚278.522' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ðlóâ/‚(:ù °•èÒŽÄ‹¦ disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 530.046875 disk.rbd3.out‚0$ memory_percent‚97.38357067108154 disk.sda1.in‚0 disk.dm-1.out‚ 530.046875( proc.memory.availableMB‚ 428.67578125# proc.memory.rssMB‚ 2822.28515625 network.recv‚ 3.473456e+06) proc.memory.percent‚17.22586154937744# disk.nvme0n1.in‚144916.93359375 disk.rbd1.in‚0.0625! disk.rbd1.out‚116.82470703125 disk.rbd3.in‚0% disk.nvme0n1.out‚54153.2412109375 disk.drbd2165.out‚0) disk./.usagePercent‚36.83043200677466$ disk./.usageGB‚548.8000526428223 disk.rbd0.in‚0' disk.nvme0c0n1.out‚54153.2412109375 disk.dm-0.in‚144916.93359375% disk.nvme0c0n1.in‚144916.93359375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 530.046875 disk.sda3.in‚0 cpu‚26.02016147653633 proc.cpu.threads‚17 network.sent‚ 2.4502106e+07" disk.dm-0.out‚54153.2412109375vç Ó“ã/‚(:ˆ ¿•èÒ¸Š§¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¢+å ä/‚(:ÿ ¿•èÒò‘¤ disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0" disk.dm-0.out‚54161.0302734375 network.recv‚ 3.476909e+06 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 network.sent‚ 2.4505028e+07# disk.nvme0n1.in‚144917.76171875 disk.sda1.out‚0 cpu‚26.01984014037716) disk./.usagePercent‚36.83046400771644$ disk./.usageGB‚548.8005294799805 disk.sda.out‚ 530.47265625 disk.sda3.out‚ 530.47265625( proc.memory.availableMB‚ 428.43359375 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚144917.76171875% disk.nvme0n1.out‚54161.0302734375 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144917.76171875' disk.nvme0c0n1.out‚54161.0302734375 disk.dm-1.out‚ 530.47265625! disk.rbd1.out‚116.82470703125 disk.rbd3.out‚0$ memory_percent‚97.38504886627197<ÌF4å// _wandbruntime‚13087  _runtime‚13087+€£:Êæ/Ä trainer/global_step‚15399 % _timestamp‚1.7842858949816144e+09 _runtime‚13087.107310664 _step‚1266 & lr-AdamW/pg1‚6.140663722502568e-05 & lr-AdamW/pg2‚6.140663722502568e-05ð&`JÏç/É _step‚1266 & lr-AdamW/pg1‚6.140663722502568e-05 & lr-AdamW/pg2‚6.140663722502568e-05 trainer/global_step‚15399 % _timestamp‚1.7842858949816144e+09 _runtime‚13087.107310664ò …Ž©õ4è// _wandbruntime‚13087  _runtime‚13087¨HŽ „é/‚P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16289008725504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16288667386368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142650289 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚531999638022656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2357159880376832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16468016860672 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚15399 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264772259 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26974641 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60127393 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2340871230029824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16288667386368 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2345092991497211 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚531999518250496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2340871212990464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515531520002048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2357160238755328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122121968 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142650289 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1267ééÀQ (ê/‡P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16468118020608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33153018 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2357160238755328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚15399 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264772259 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26974641 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60127393 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2340871230029824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16288667386368 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2345092991497211 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚531999518250496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2340871212990464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515531520002048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2357160238755328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122121968 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142650289 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1267 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16289008725504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16288667386368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142650289 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚531999638022656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2357159880376832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16468016860672 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 BgÄ 'ì/‹N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2341018471059456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60131433 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚532034085072896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2357308492419072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16469136312320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16290021359616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2328975326412800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122129600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16289680013312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122129600 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 % _timestamp‚1.7842858955934596e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142659355 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26976386 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532033979987968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16265169770757 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16469049839616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515564948760576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142649364 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33155047 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2345240513222917 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122129602 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16290021359616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2341018454020096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60131167 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚1268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 264788957 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264778964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2341018454020096 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264788957 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142659355 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264778964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2357308492419072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16289680013312 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264778964 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2357308492419072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16289680013312 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16290021359616 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2328975326412800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122129600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16289680013312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122129600 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 % _timestamp‚1.7842858955934596e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142659355 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26976386 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532033979987968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16265169770757 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16469049839616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515564948760576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_ÎÐq bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142649364 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33155047 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2345240513222917 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122129602 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16290021359616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2341018454020096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60131167 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚1268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 264788957 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264778964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2341018454020096 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264788957 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142659355ô ,­¼$4î// _wandbruntime‚13087  _runtime‚13087u˜gÀï/‰ 0loss/FRt2/assignment_loss‚0.7809926058039253 4loss/FBt2/assignment_loss‚1.1301821275289115e-05 /loss/FRt2/detection_loss‚1.1974315157982358 1loss/SRqqt2/detection_loss‚0.5817153142753946 $loss/total_loss‚7.04447402802335 trainer/global_step‚15399 /loss/FRt1/assignment_loss‚1.410150364109061 3loss/SRqqt2/assignment_loss‚0.43709495544255567 /loss/FRt1/detection_loss‚0.5187094671731839 % _timestamp‚1.7842858955970592e+09 _runtime‚13087.722409732 3loss/FBt1/assignment_loss‚0.0021505427932941193 1loss/SRqqt1/detection_loss‚0.8877390833416333 /loss/FBt1/detection_loss‚0.3724826842915636 2loss/SRqqt1/assignment_loss‚0.2616470609918489 /loss/FBt2/detection_loss‚0.5943491321813781 epoch‚38 _step‚1269+‹#Åð/Ž 0loss/FRt2/assignment_loss‚0.7809926058039253 4loss/FBt2/assignment_loss‚1.1301821275289115e-05 /loss/FRt2/detection_loss‚1.1974315157982358 1loss/SRqqt2/detection_loss‚0.5817153142753946 $loss/total_loss‚7.04447402802335 trainer/global_step‚15399 /loss/FRt1/assignment_loss‚1.410150364109061 3loss/SRqqt2/assignment_loss‚0.43709495544255567 /loss/FRt1/detection_loss‚0.5187094671731839 % _timestamp‚1.7842858955970592e+09 _runtime‚13087.722409732 3loss/FBt1/assignment_loss‚0.0021505427932941193 1loss/SRqqt1/detection_loss‚0.8877390833416333 /loss/FBt1/detection_loss‚0.3724826842915636 2loss/SRqqt1/assignment_loss‚0.2616470609918489 /loss/FBt2/detection_loss‚0.5943491321813781 epoch‚38 _step‚1269õ Fàö“ñ/‚(:ˆ ΕèÒ•Ò¼¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚244.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.9678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øœ”ò/‚(:ù ΕèÒõί¦* disk./.usagePercent‚36.830468359844524 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 530.609375$ memory_percent‚97.38385677337646" disk.nvme0n1.in‚144918.4765625 disk.sda3.in‚0! disk.rbd1.out‚116.88720703125 disk.sda.out‚ 530.609375( proc.memory.availableMB‚ 428.62890625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚144918.4765625 disk.rbd3.in‚0) proc.memory.percent‚17.22586154937744 cpu‚26.019448358853438 proc.cpu.threads‚17& disk.nvme0n1.out‚54168.04248046875# disk.dm-0.out‚54168.04248046875 network.sent‚ 2.451008e+07 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0# disk./.usageGB‚548.800594329834 disk.rbd0.in‚0 disk.dm-0.in‚144918.4765625 network.recv‚ 3.479917e+06 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚54168.04248046875 disk.sda3.out‚ 530.609375# proc.memory.rssMB‚ 2822.28515625Ri•žó/‚(:“ Ý•èÒçè·¢ gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CXÒ%ô/‚(:ü Ý•èÒÄäˤ network.recv‚ 3.484687e+06 disk.rbd3.out‚0 network.sent‚ 2.4573202e+07 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 530.8359375& disk.nvme0n1.out‚54173.11474609375 disk.sda1.out‚0$ memory_percent‚97.38650321960449' proc.memory.availableMB‚ 428.1953125 cpu‚26.019091462831337 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚144918.4765625" disk.nvme0n1.in‚144918.4765625 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744( disk.nvme0c0n1.out‚54173.11474609375 disk.sda3.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚116.88720703125 disk.drbd2165.out‚0) disk./.usagePercent‚36.83046989588972 disk.sda3.out‚ 530.8359375 disk.rbd1.in‚0.0625$ disk./.usageGB‚548.8006172180176$ disk.nvme0c0n1.in‚144918.4765625 disk.dm-1.out‚ 530.8359375# disk.dm-0.out‚54173.11474609375jµmŽõ/‚(:’ ì•èÒœ³Ø¢ gpu.0.gpu‚20.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚205.224' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.044799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z…†ö/‚(:ù ì•èÒ¨¥Ý¤ disk.rbd0.out‚0 disk.sda.out‚ 530.9609375% disk.nvme0n1.out‚54181.5126953125 cpu‚26.0187654450601 network.recv‚ 3.487707e+06 disk.sda.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 428.44140625 proc.cpu.threads‚17 disk.sda3.out‚ 530.9609375 disk.sda1.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚144919.3046875' disk.nvme0c0n1.out‚54181.5126953125# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 disk.sda2.in‚0$ memory_percent‚97.38500118255615$ disk./.usageGB‚548.8005256652832 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 530.9609375 disk.rbd3.in‚0 network.sent‚ 2.4578254e+07 disk.rbd1.in‚0.0625! disk.rbd1.out‚116.94970703125 disk.rbd3.out‚0" disk.nvme0n1.in‚144919.3046875 disk.rbd0.in‚0* disk./.usagePercent‚36.830463751708905 disk.sda3.in‚0 disk.dm-0.in‚144919.3046875" disk.dm-0.out‚54181.5126953125 disk.drbd2165.in‚0p[yë4÷// _wandbruntime‚13127  _runtime‚13127c —Éø/à & lr-AdamW/pg1‚6.013867587817664e-05 & lr-AdamW/pg2‚6.013867587817664e-05 trainer/global_step‚15449 % _timestamp‚1.7842859348871658e+09 _runtime‚13127.01277021 _step‚1270Õçs}Îù/È & lr-AdamW/pg1‚6.013867587817664e-05 & lr-AdamW/pg2‚6.013867587817664e-05 trainer/global_step‚15449 % _timestamp‚1.7842859348871658e+09 _runtime‚13127.01277021 _step‚1270ö ðéX4ú// _wandbruntime‚13127  _runtime‚13127 óû(û/€P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚517202939316224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2352468783988469 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2348233264472064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33256121 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2364572985738752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16339362888192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2364572985738752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2336154366373888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2364572627360256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚1271 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143094764 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2348233281511424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265608311 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16339362888192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33256390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265598332 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2348233281511424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122503570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2336154349334528 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16519148175872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143094764 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2364572627360256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27061888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚15449 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143104741 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533722087492096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533722194681344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169ݾn´ (ü/…P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122503568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143104741 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16339704227328 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60318281 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122503568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16339362888192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33256390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265598332 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2348233281511424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122503570 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2336154349334528 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16519148175872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143094764 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2364572627360256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27061888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚15449 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143104741 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533722087492096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533722194681344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚517202939316224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2352468783988469 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2348233264472064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33256121 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2364572985738752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16339362888192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2364572985738752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2336154366373888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2364572627360256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚1271 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143094764 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2348233281511424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265608311 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122511202 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2348380505501696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27063633 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2364721239402496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16340375515136 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2348380505501696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60321802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27063636 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚517236368074752 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16315445707663 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60322053 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2336300860006400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143103816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16520161358336 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265625009 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143103816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33258417 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16520254122496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2364721239402496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16340716861440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚517236386686976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122511202 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2336300877045760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60321802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27063636 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚517236368074752 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16315445707663 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60322053 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2336300860006400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143103816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16520161358336 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265625009 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143103816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33258417 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16520254122496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2364721239402496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16340716861440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚517236386686976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122511202 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2336300877045760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122511202 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2348380505501696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27063633 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2364721239402496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16340375515136 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2348380505501696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2371985732722176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚15499 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2355595332993024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2359844934068977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122885170 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60509390 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27149138 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2371985374343680 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2355595315953664 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16390399729152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143549216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266444363 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266434384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143549216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16365391690739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16390058390016 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2359845291658227 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2371985732722176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143559193 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2355595315953664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2343479899967488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16570184908288 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚13166.967388343 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143559193 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚518874377242624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535444562150912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2343479882928128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842859748377874e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122885168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122885170 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2371985374343680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33360252 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.oYkˆ)n_train_batch_start/num_alloc_retries‚0_î (Œ0†P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143549216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16365391690739 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16390058390016 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2359845291658227 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2371985732722176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143559193 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2355595315953664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2343479899967488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16570184908288 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚13166.967388343 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143559193 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚518874377242624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535444562150912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2343479882928128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842859748377874e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122885168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122885170 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2371985374343680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33360252 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122885168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535444669340160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2371985732722176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚15499 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2355595332993024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2359844934068977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122885170 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60509390 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27149138 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2371985374343680 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2355595315953664 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16390399729152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143549216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266444363 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266434384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0û )CÁì40/ _wandbruntime‚13167  _runtime‚13167çO×°'Ž0ŠN XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33362237 % _timestamp‚1.7842859754470422e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122892800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16571298849792 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2355742574022656 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122892800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2343626410639360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60513120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚518907824613376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16391071016960 _step‚1276 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2355742556983296 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2372133628000256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60513368 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16391071016960 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122892802 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16366402744573 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143568259 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27150883 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33362482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266461061 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16366062194571 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143558268 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143558268uôãÚ'0N SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2372133628000256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60513368 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16391071016960 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122892802 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16366402744573 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143568259 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27150883 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33362482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266461061 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16366062194571 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143558268 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143558268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33362237 % _timestamp‚1.7842859754470422e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122892800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16571298849792 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2355742574022656 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122892800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2343626410639360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60513120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚518907824613376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16391071016960 _step‚1276 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2355742556983296ü RðPJ40/ _wandbruntime‚13167  _runtime‚13167Ø·Ž‘0‹ /loss/FBt2/detection_loss‚0.4832941682834644 epoch‚38 $ _timestamp‚1.784285975450979e+09 _runtime‚13167.576327234 4loss/FBt2/assignment_loss‚3.9187855412550604e-05 /loss/FRt1/detection_loss‚0.5037314552379868 1loss/SRqqt1/detection_loss‚0.8295441243940331 trainer/global_step‚15499 0loss/FRt1/assignment_loss‚1.4338490159993762 4loss/FBt1/assignment_loss‚0.00040903271610006356 1loss/SRqqt2/detection_loss‚0.6228707380982264 _step‚1277 %loss/total_loss‚6.915162218060328 0loss/FRt2/assignment_loss‚0.7264416601734229 2loss/SRqqt1/assignment_loss‚0.3227011252543118 2loss/SRqqt2/assignment_loss‚0.4520172109233926 /loss/FRt2/detection_loss‚1.1196387613845897 0loss/FBt1/detection_loss‚0.42062573774001066ÚK3ù’0 /loss/FRt1/detection_loss‚0.5037314552379868 1loss/SRqqt1/detection_loss‚0.8295441243940331 trainer/global_step‚15499 0loss/FRt1/assignment_loss‚1.4338490159993762 4loss/FBt1/assignment_loss‚0.00040903271610006356 1loss/SRqqt2/detection_loss‚0.6228707380982264 _step‚1277 %loss/total_loss‚6.915162218060328 0loss/FRt2/assignment_loss‚0.7264416601734229 2loss/SRqqt1/assignment_loss‚0.3227011252543118 2loss/SRqqt2/assignment_loss‚0.4520172109233926 /loss/FRt2/detection_loss‚1.1196387613845897 0loss/FBt1/detection_loss‚0.42062573774001066 /loss/FBt2/detection_loss‚0.4832941682834644 epoch‚38 $ _timestamp‚1.784285975450979e+09 _runtime‚13167.576327234 4loss/FBt2/assignment_loss‚3.9187855412550604e-05ý æö¿`““0‚(:ˆ ™–èÒìÏÿ¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”Ô1Â>”0‚(:ê ™–èÒóíߧ# disk.dm-0.out‚54207.54248046875 disk.rbd0.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚54207.54248046875 disk.drbd2165.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚531.9375 disk.sda1.in‚0$ memory_percent‚97.38481044769287Ä”þ· cpu‚26.01774321859873 network.sent‚ 2.4664507e+07 disk.sda3.out‚531.9375! disk.rbd1.out‚117.01220703125 disk.rbd3.in‚0 disk.dm-0.in‚ 144921.5625 disk.sda1.out‚0( disk.nvme0c0n1.out‚54207.54248046875( proc.memory.availableMB‚ 428.47265625 disk.sda.in‚0 disk.sda.out‚531.9375 network.recv‚ 3.498968e+06# disk./.usageGB‚548.800594329834 disk.sda2.out‚0# proc.memory.rssMB‚ 2822.28515625* disk./.usagePercent‚36.830468359844524 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 144921.5625 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 144921.5625) proc.memory.percent‚17.22586154937744 disk.dm-1.in‚0 disk.rbd3.out‚0_Ï2œ•0‚(:‘ ¨–èÒ⪾¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.46' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.69200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÜY=–0‚(:ö ¨–èÒæÎé¤ disk.sda2.out‚0 network.sent‚ 2.4713581e+07 cpu‚26.017376156116285$ disk./.usageGB‚548.7994537353516 disk.dm-1.in‚0 disk.rbd1.in‚0.0625( disk.nvme0c0n1.out‚54210.75341796875 disk.sda3.in‚0! disk.rbd1.out‚117.07470703125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚54210.75341796875 disk.sda.in‚0 network.recv‚ 3.50273e+06 disk.rbd0.out‚0 disk.sda.out‚ 532.05859375$ memory_percent‚97.38352298736572 disk.dm-1.out‚ 532.05859375& disk.nvme0n1.out‚54210.75341796875 disk.sda1.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 disk.sda3.out‚ 532.05859375 disk.dm-0.in‚ 144921.5625 disk.nvme0n1.in‚ 144921.5625( proc.memory.availableMB‚ 428.68359375 proc.cpu.threads‚17) disk./.usagePercent‚36.83039181359178 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 144921.5625 disk.sda2.in‚0Ì÷øG“—0‚(:ˆ ·–èÒÇ—¿¢ gpu.0.gpu‚94.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚232.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.4342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Tк˜0‚(:ý ·–èÒÝæÖ¤* disk./.usagePercent‚36.830402821915754 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 144922.390625$ memory_percent‚97.38352298736572) proc.memory.percent‚17.22586154937744# disk./.usageGB‚548.799617767334! disk.nvme0n1.in‚ 144922.390625 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚54220.06103515625( proc.memory.availableMB‚ 428.68359375# proc.memory.rssMB‚ 2822.28515625 cpu‚26.017073958326264 network.recv‚ 3.506039e+06 disk.rbd1.in‚0.0625! disk.rbd1.out‚117.07470703125& disk.nvme0n1.out‚54220.06103515625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 2.4720712e+07 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 532.48828125 disk.sda2.out‚0 disk.sda3.out‚ 532.48828125 disk.dm-1.out‚ 532.48828125 disk.sda1.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 144922.390625# disk.dm-0.out‚54220.06103515625 disk.drbd2165.in‚0vIùß4™0/ _wandbruntime‚13206  _runtime‚13206‰&øsÊš0Ä & lr-AdamW/pg1‚5.763678400628776e-05 & lr-AdamW/pg2‚5.763678400628776e-05 trainer/global_step‚15549 % _timestamp‚1.7842860145752823e+09 _runtime‚13206.701066753 _step‚12783b‘Ï›0É trainer/global_step‚15549 % _timestamp‚1.7842860145752823e+09 _runtime‚13206.701066753 _step‚1278 & lr-AdamW/pg1‚5.763678400628776e-05 & lr-AdamW/pg2‚5.763678400628776e-05þ +xÞ‹4œ0/ _wandbruntime‚13206  _runtime‚13206 ¾Å(0ÿO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚537167123965440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16621403664896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2350805416521728 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16415667627757 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144013645 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2379398121327104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16441095230976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚13206.702430528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2379398121327104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33463271 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚520545815169024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 # _timestamp‚1.78428601457667e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2367221084149485 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267280415 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚537167237446144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2362957367435264 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2379398479705600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144003668 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123266770 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2350805433561088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123266768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16621308796416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚15549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123266768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27236391 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2362957384474624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2379398121327104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33463271 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚520545815169024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 # _timestamp‚1.78428601457667e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2367221084149485 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267280415 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚537167237446144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2362957367435264 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2379398479705600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144003668 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123266770 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2350805433561088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123266768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16621308796416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚15549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123266768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27236391 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2362957384474624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169ÿ ×Ù1æ4Ÿ0/ _wandbruntime‚13207  _runtime‚13207…+*¦r 0ŠN $ _timestamp‚1.784286015187488e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2350951944232960 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144012720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27238133 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123274402 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267287120 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33465501 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2367368605875079 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2379546733369344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16441766518784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60703372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16417019231481 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123274400 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123274400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2350951927193600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27238136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144012720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2363104608464896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60703637 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚537201687834624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚537201582749696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2367368963464441 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2379546733369344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚15549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16442107865088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267297113 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144022711 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16442107865088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520579262539776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16622425294848 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2363104625504256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0òëNÜ'¡0N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33465501 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2367368605875079 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2379546733369344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16441766518784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60703372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16417019231481 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123274400 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123274400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2350951927193600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27238136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144012720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2363104608464896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60703637 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚537201687834624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚537201582749696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2367368963464441 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2379546733369344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚15549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16442107865088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267297113 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144022711 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16442107865088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520579262539776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16622425294848 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2363104625504256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784286015187488e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2350951944232960 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144012720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27238133 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123274402 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267287120 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169€ ZiHk4¢0/ _wandbruntime‚13207  _runtime‚13207/T÷ £0† /loss/FBt2/detection_loss‚0.6323006182376828 epoch‚38 trainer/global_step‚15549 _step‚1281 3loss/SRqqt1/assignment_loss‚0.33153274419984796 2loss/FBt2/assignment_loss‚0.015352939965131793 /loss/FRt2/detection_loss‚1.1188114607080384 1loss/FBt1/assignment_loss‚0.01077245326597389 1loss/SRqqt1/detection_loss‚0.9808819160852138 $ _timestamp‚1.784286015190666e+09 _runtime‚13207.315998734 0loss/FRt1/assignment_loss‚1.5466780043690582 0loss/FRt2/assignment_loss‚0.7401621606956232 1loss/SRqqt2/detection_loss‚0.5729529200610766 %loss/total_loss‚7.230228107022995 2loss/SRqqt2/assignment_loss‚0.4427516025488943 /loss/FRt1/detection_loss‚0.4910110132032169 /loss/FBt1/detection_loss‚0.3470202736832378ÆÀ_P¤0‹ 3loss/SRqqt1/assignment_loss‚0.33153274419984796 2loss/FBt2/assignment_loss‚0.015352939965131793 /loss/FRt2/detection_loss‚1.1188114607080384 1loss/FBt1/assignment_loss‚0.01077245326597389 1loss/SRqqt1/detection_loss‚0.9808819160852138 $ _timestamp‚1.784286015190666e+09 _runtime‚13207.315998734 0loss/FRt1/assignment_loss‚1.5466780043690582 0loss/FRt2/assignment_loss‚0.7401621606956232 1loss/SRqqt2/detection_loss‚0.5729529200610766 %loss/total_loss‚7.230228107022995 2loss/SRqqt2/assignment_loss‚0.4427516025488943 /loss/FRt1/detection_loss‚0.4910110132032169 /loss/FBt1/detection_loss‚0.3470202736832378 /loss/FBt2/detection_loss‚0.6323006182376828 epoch‚38 trainer/global_step‚15549 _step‚1281 ­š£#‘¥0‚(:† Æ–èÒõ‹¥£ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.94' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<|[ž ¦0‚(:þ Æ–èÒ§Ó“¥ disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625& disk.nvme0n1.out‚54226.17529296875# proc.memory.rssMB‚ 2822.28515625( disk.nvme0c0n1.out‚54226.17529296875# disk.nvme0n1.in‚144923.10546875! disk.rbd1.out‚117.13720703125$ proc.memory.availableMB‚428.3125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚144923.10546875 network.recv‚ 3.509282e+06$ disk./.usageGB‚548.7907180786133 disk.drbd2165.in‚0 disk.sda3.out‚ 532.67578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚0% disk.nvme0c0n1.in‚144923.10546875 disk.sda.out‚ 532.67578125 disk.dm-1.in‚0 disk.dm-1.out‚ 532.67578125) proc.memory.percent‚17.22586154937744) disk./.usagePercent‚36.82980555633837 disk.rbd3.out‚0# disk.dm-0.out‚54226.17529296875$ memory_percent‚97.38578796386719 cpu‚26.01676989766444 network.sent‚ 2.4723634e+07ñü®‘§0‚(:† Õ–èÒ„Žß¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vä4Òú¨0‚(:ï Õ–èÒ†ž†¥ disk.dm-0.out‚ 54235.96875 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 532.8515625 disk.sda2.in‚0 disk.sda3.in‚0 network.sent‚ 2.4786994e+07% disk.nvme0c0n1.in‚144923.93359375$ disk./.usageGB‚548.7906684875488! disk.rbd1.out‚117.13720703125 disk.sda1.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.829802228240425 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.dm-0.in‚144923.93359375$ memory_percent‚97.38576412200928 disk.drbd2165.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 428.31640625 network.recv‚ 3.514053e+06 disk.rbd3.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 cpu‚26.016431243315264 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.out‚ 54235.96875 disk.dm-1.out‚ 532.8515625 disk.nvme0n1.out‚ 54235.96875# disk.nvme0n1.in‚144923.93359375 disk.sda3.out‚ 532.8515625 disk.dm-1.in‚0ƒtlî“©0‚(:ˆ ä–èÒ™ùç£ gpu.0.gpu‚73.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚA¦=ª0‚(:ü ä–èÒ“–’¥" disk.nvme0n1.in‚144924.6484375) disk./.usagePercent‚36.82979838812741 disk.rbd0.in‚0) proc.memory.percent‚17.22586154937744 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚144924.6484375( proc.memory.availableMB‚ 428.31640625 cpu‚26.01609257464624 disk.sda2.in‚0 disk.sda3.in‚0" disk.dm-0.out‚54243.3798828125 network.sent‚ 2.4792006e+07 disk.sda1.out‚0' disk.nvme0c0n1.out‚54243.3798828125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2822.28515625$ disk./.usageGB‚548.7906112670898$ disk.nvme0c0n1.in‚144924.6484375 disk.sda3.out‚ 533.04296875 disk.rbd1.in‚0.0625! disk.rbd1.out‚117.19970703125 proc.cpu.threads‚17 disk.sda2.out‚0$ memory_percent‚97.38576412200928% disk.nvme0n1.out‚54243.3798828125 network.recv‚ 3.517234e+06 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 533.04296875 disk.sda.out‚ 533.04296875 disk.rbd3.out‚0ü$–4«0/ _wandbruntime‚13246  _runtime‚132463X–ʬ0Ä & lr-AdamW/pg1‚5.640301417308742e-05 & lr-AdamW/pg2‚5.640301417308742e-05 trainer/global_step‚15599 % _timestamp‚1.7842860544747016e+09 _runtime‚13246.600413934 _step‚1282·È Ï­0É & lr-AdamW/pg1‚5.640301417308742e-05 & lr-AdamW/pg2‚5.640301417308742e-05 trainer/global_step‚15599 % _timestamp‚1.7842860544747016e+09 _runtime‚13246.600413934 _step‚1282‚ EÍÂ4®0/ _wandbruntime‚13246  _runtime‚13246AcD(¯0P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27323641 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522217253095424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16672315520512 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2374597234229993 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123648368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16491790732800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33566271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚538889673708032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚13246.601616114 _step‚1283 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144458120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60889631 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16466284114665 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2370319418916864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2386810868310528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2370319418916864 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16466624664555 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚15599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2374597591819243 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16491790732800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60889912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123648370 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2358130950115328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33565993 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16672402000384 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123648368 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2386811226689024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27323638 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0[ˆ[©¾°0†P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚13246.601616114 _step‚1283 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144458120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60889631 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16466284114665 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2370319418916864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2386810868310528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2370319418916864 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16466624664555 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚15599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2374597591819243 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16491790732800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60889912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123648370 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2358130950115328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33565993 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16672402000384 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123648368 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2386811226689024 TGDeviceStatsMonitor.on_train_batch_s‚‡ãøNtart/inactive_split.large_pool.freed‚27323638 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27323641 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522217253095424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16672315520512 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2374597234229993 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123648368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16491790732800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33566271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚538889673708032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ƒ Q,CÃ4±0/ _wandbruntime‚13247  _runtime‚13247¤ÀûY'²0ŠN \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2370466676985856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2370466659946496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16492803366912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268123172 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144467172 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268133165 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27325383 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144477163 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123656002 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123656002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27325386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123656000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2386959480352768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2374745113544949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚13247.214475857 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16492462020608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚538924016496640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60893390 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2374744755955587 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268133165 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33568007 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16492462020608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268123172 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33568260 _step‚1284 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144477163 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚522250700466176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2386959121967104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚15599 $ _timestamp‚1.784286055088615e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16467635718389 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚538924125775872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144467172 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2358277477826560 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2358277460787200 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2386959480352768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16673425309696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144467172 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268133165 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27325383 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144477163 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123656002 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123656002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27325386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123656000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2386959480352768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2374745113544949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚13247.214475857 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16492462020608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚538924016496640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60893390 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2374744755955587 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268133165 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33568007 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16492462020608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268123172 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33568260 _step‚1284 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144477163 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚522250700466176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2386959121967104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚15599 $ _timestamp‚1.784286055088615e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16467635718389 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0„ d—4´0/ _wandbruntime‚13247  _runtime‚13247ó¥Ñµ0ˆ _step‚1285 0loss/FRt1/assignment_loss‚1.5480476400643721 2loss/FBt1/assignment_loss‚0.015979009343664405 /loss/FRt1/detection_loss‚0.5481128693496683 %loss/total_loss‚7.154037440980302 0loss/FRt2/assignment_loss‚0.7167227459610055 /loss/FRt2/detection_loss‚1.2698432254667475 1loss/SRqqt1/detection_loss‚1.0027694326810888 0loss/FBt1/detection_loss‚0.36762511498176537 2loss/SRqqt2/assignment_loss‚0.4451690070034115 3loss/FBt2/assignment_loss‚7.413298963587778e-07 1loss/SRqqt2/detection_loss‚0.6056598087695116 trainer/global_step‚15599 % _timestamp‚1.7842860550911636e+09 1loss/SRqqt1/assignment_loss‚0.337868692168801 /loss/FBt2/detection_loss‚0.2962391538603697 epoch‚38 _runtime‚13247.216528503ýÊÔ]¶0 0loss/FRt1/assignment_loss‚1.5480476400643721 2loss/FBt1/assignment_loss‚0.015979009343664405 /loss/FRt1/detection_loss‚0.5481128693496683 %loss/total_loss‚7.154037440980302 0loss/FRt2/assignment_loss‚0.7167227459610055 /loss/FRt2/detection_loss‚1.2698432254667475 1loss/SRqqt1/detection_loss‚1.0027694326810888 0loss/FBt1/detection_loss‚0.36762511498176537 2loss/SRqqt2/assignment_loss‚0.4451690070034115 3loss/FBt2/assignment_loss‚7.413298963587778e-07 1loss/SRqqt2/detection_loss‚0.6056598087695116 trainer/global_step‚15599 % _timestamp‚1.7842860550911636e+09 1loss/SRqqt1/assignment_loss‚0.337868692168801 /loss/FBt2/detection_loss‚0.2962391538603697 epoch‚38 _runtime‚13247.216528503 _step‚1285… ñ?ý‘·0‚(:† ó–èÒËÖ½¢ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0߯ˆY¸0‚(:ý ó–èҊ͈¥ disk.sda2.out‚0# proc.memory.rssMB‚ 2822.28515625) disk./.usagePercent‚36.83031603536164$ disk./.usageGB‚548.7983245849609 disk.sda1.in‚0$ disk.nvme0c0n1.in‚144924.6484375 disk.sda.in‚0 disk.sda3.out‚ 533.3359375 disk.rbd3.in‚0" disk.nvme0n1.in‚144924.6484375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚54249.30810546875$ memory_percent‚97.38719463348389! disk.rbd1.out‚117.19970703125 disk.dm-0.in‚144924.6484375# disk.dm-0.out‚54249.30810546875 disk.drbd2165.in‚0 disk.sda.out‚ 533.3359375 disk.sda2.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚54249.30810546875 disk.dm-1.out‚ 533.3359375 disk.sda3.in‚0 disk.dm-1.in‚0 cpu‚26.019344041774794 proc.cpu.threads‚17 disk.drbd2165.out‚0( proc.memory.availableMB‚ 428.08203125) proc.memory.percent‚17.22586154937744 disk.rbd1.in‚0.0625 network.sent‚ 2.4796898e+07 network.recv‚ 3.520702e+06××pÚ4¹0/ _wandbruntime‚13264  _runtime‚13264†QÉwê º0ïz :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7673245668411255 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.038940172642469406 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2995204031467438 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9896690845489502 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6661669611930847 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2588716745376587 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7532501816749573 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.020900584757328033 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.022031640633940697 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24040257930755615 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2624470293521881 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8726589679718018 7validation_average_jet_accuracy‚0.37582218647003174 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.06481482088565826 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_2_of_2‚0.16066578030586243 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6357806921005249 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6625228524208069 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.364398330450058 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7514927983283997 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022697098553180695 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2076125144958496 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20436079800128937 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04630912467837334 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6453250050544739 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8726589679718018 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2494076043367386 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32034245133399963ÞT‚‹0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964953660964966 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7970808148384094 *jet/accuracy_1_of_4‚0.9817572236061096 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24451787769794464 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879798889160156 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7291324734687805 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27287301421165466 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916567325592 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7291452288627625 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7476568818092346 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22601813077926636 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8494237065315247 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6285353302955627 *jet/accuracy_1_of_2‚0.7929648160934448 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6982011795043945 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5081657767295837 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6076123714447021 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23907582461833954 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26925066113471985 jet/accuracy_3_of_5‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.601280927658081 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9009417295455933 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23995763063430786 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5847775340080261 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46700701117515564 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 _step‚1286 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5725551247596741 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9220192432403564 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_3_of_3‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6066256165504456 +particle/sensitivity‚0.6136094331741333 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6150027513504028 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07388567179441452 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.695061981678009 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7532501816749573 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911108613014221 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5029353499412537 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03768591582775116 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2632848024368286 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.06281380355358124 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9956878423690796 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8605564832687378 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7386999130249023 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6358803510665894 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 /particle/accuracy_4_of_4‚0.9482209086418152 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6645060181617737 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19861538708209991 *particle/specificity‚0.905180811882019 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2744601368904114 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0459199883043766 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6377730369567871 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8425641655921936 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9675925374031067 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9898518919944763 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7741950154304504 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5482503175735474 *jet/accuracy_1_of_3‚0.9247206449508667 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6322996020317078 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6677457690238953 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08741671591997147 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2853115499019623 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6319800019264221 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6233842372894287 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 +jet/accuracy_3_of_4‚0.07847467809915543 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6150027513504028 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23805774748325348 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7201834917068481 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6883409023284912 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38769471645355225 +jet/accuracy_1_of_1‚0.33249419927597046 *jet/accuracy_2_of_4‚0.8048148155212402 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21210868656635284 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993890106678009 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6680122017860413 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2624470293521881 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976133704185486 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6428622007369995 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022697098553180695 +jet/accuracy_4_of_4‚0.00957049522548914 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.06481482088565826 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21833841502666473 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 'particle/f_score‚0.6260092854499817 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2543906569480896 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3649493157863617 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27313026785850525 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3337894082069397 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5166060328483582 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 _runtime‚13264.280853805 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 6"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7633096575737 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7652490735054016 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6639059782028198 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6374670267105103 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.271965354681015 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 epoch‚38 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6078569293022156 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5326215028762817 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.020712055265903473 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7842860721546474e+09 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5821532011032104 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_2_of_4‚1 *loss/val_total_loss‚7.9762833370690815 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021882113069295883 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22641754150390625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.654185950756073 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946811199188232 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2504471242427826 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 trainer/global_step‚15599 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7913764119148254 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 particle/accuracy_5_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9898712635040283 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23943674564361572 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6132638454437256 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5433333516120911 *jet/accuracy_2_of_3‚0.5752947330474854:::z=»0ôz 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7673245668411255 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.038940172642469406 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2995204031467438 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9896690845489502 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6661669611930847 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2588716745376587 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7532501816749573 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.020900584757328033 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.022031640633940697 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24040257930755615 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2624470293521881 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8726589679718018 7validation_average_jet_accuracy‚0.37582218647003174 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.06481482088565826 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_2_of_2‚0.16066578030586243 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6357806921005249 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6625228524208069 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.364398330450058 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7514927983283997 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022697098553180695 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2076125144958496 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20436079800128937 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04630912467837334 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6453250050544739 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8726589679718018 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2494076043367386 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32034245133399963 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964953660964966 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7970808148384094 *jet/accuracy_1_of_4‚0.9817572236061096 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24451787769794464 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879798889160156 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7291324734687805 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27287301421165466 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916567325592 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7291452288627625 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7476568818092346 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22601813077926636 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8494237065315247 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6285353302955627 *jet/accuracy_1_of_2‚0.7929648160934448 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6982011795043945 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5081657767295837 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6076123714447021 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23907582461833954 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26925066113471985 jet/accuracy_3_of_5‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.601280927658081 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9009417295455933 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23995763063430786 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5847775340080261 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46700701117515564 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 _step‚1286 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5725551247596741 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9220192432403564 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_3_of_3‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6066256165504456 +particle/sensitivity‚0.6136094331741333 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6150027513504028 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07388567179441452 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.695061981678009 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7532501816749573 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911108613014221 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5029353499412537 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03768591582775116 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2632848024368286 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.06281380355358124 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9956878423690796 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8605564832687378 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7386999130249023 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6358803510665894 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 /particle/accuracy_4_of_4‚0.9482209086418152 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6645060181617737 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19861538708209991 *particle/specificity‚0.905180811882019 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2744601368904114 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0459199883043766 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6377730369567871 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8425641655921936 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9675925374031067 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9898518919944763 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7741950154304504 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5482503175735474 *jet/accuracy_1_of_3‚0.9247206449508667 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6322996020317078 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6677457690238953 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08741671591997147 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2853115499019623 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6319800019264221 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6233842372894287 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 +jet/accuracy_3_of_4‚0.07847467809915543 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6150027513504028 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23805774748325348 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7201834917068481 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6883409023284912 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38769471645355225 +jet/accuracy_1_of_1‚0.33249419927597046 *jet/accuracy_2_of_4‚0.8048148155212402 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21210868656635284 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993890106678009 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6680122017860413 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2624470293521881 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976133704185486 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6428622007369995 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022697098553180695 +jet/accuracy_4_of_4‚0.00957049522548914 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.06481482088565826 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21833841502666473 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 'particle/f_score‚0.6260092854499817 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2543906569480896 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3649493157863617 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27313026785850525 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3337894082069397 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5166060328483582 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 _runtime‚13264.280853805 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 6"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7633096575737 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7652490735054016 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6639059782028198 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6374670267105103 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.271965354681015 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 epoch‚38 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6078569293022156 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5326215028762817 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.020712055265903473 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7842860721546474e+09 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5821532011032104 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_2_of_4‚1 *loss/val_total_loss‚7.9762833370690815 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021882113069295883 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22641754150390625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.654185950756073 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946811199188232 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2504471242427826 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 trainer/global_step‚15599 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7913764119148254 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 particle/accuracy_5_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 jet/accuracy_2_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9898712635040283 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23943674564361572 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6132638454437256 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5433333516120911 *jet/accuracy_2_of_3‚0.5752947330474854 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484† ¯Ý°ªž¼0‚(:“ ‚—èÒ«¦·¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_¶*3 ½0‚(:ÿ ‚—èÒìÍœ¦) proc.memory.percent‚17.22586154937744 cpu‚26.020481109352342 network.recv‚ 3.52483e+06$ disk.nvme0c0n1.in‚144925.4765625 disk.dm-1.in‚0) disk./.usagePercent‚36.83030681909042$ disk./.usageGB‚548.7981872558594 disk.rbd0.in‚0 disk.sda3.out‚ 533.59765625# proc.memory.rssMB‚ 2822.28515625 disk.rbd0.out‚0$ memory_percent‚97.38690853118896 disk.drbd2165.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚144925.4765625& disk.nvme0n1.out‚54256.23583984375( proc.memory.availableMB‚ 428.12890625# disk.dm-0.out‚54256.23583984375 disk.sda1.in‚0( disk.nvme0c0n1.out‚54256.23583984375 disk.dm-1.out‚ 533.59765625! disk.rbd1.out‚117.26220703125 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda3.in‚0 network.sent‚ 2.4860056e+07 disk.sda.out‚ 533.59765625 disk.rbd1.in‚0.0625 disk.dm-0.in‚144925.4765625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0]“¾0‚(:ˆ ‘—èÒÁ¸¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.828' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢¨¿0‚(:ù ‘—èÒÁ™å¤$ memory_percent‚97.38690853118896 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 533.78125 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚144926.19140625# disk.dm-0.out‚54265.29345703125 disk.sda1.in‚0% disk.nvme0c0n1.in‚144926.19140625) proc.memory.percent‚17.22586154937744 cpu‚26.020145239547546 disk.sda1.out‚0 disk.sda.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 428.12890625 network.recv‚ 3.52913e+06) disk./.usagePercent‚36.83033113980617 disk.sda3.out‚ 533.78125 disk.dm-1.out‚ 533.78125& disk.nvme0n1.out‚54265.29345703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚54265.29345703125 proc.cpu.threads‚17# disk.nvme0n1.in‚144926.19140625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2822.28515625 network.sent‚ 2.4917396e+07$ disk./.usageGB‚548.7985496520996! disk.rbd1.out‚117.26220703125fÌY×4À0/ _wandbruntime‚13303  _runtime‚13303¢{ÔôÌÁ0Æ _step‚1287 ' lr-AdamW/pg1‚5.5180800977748514e-05 ' lr-AdamW/pg2‚5.5180800977748514e-05 trainer/global_step‚15649 % _timestamp‚1.7842861115227394e+09 _runtime‚13303.648370714ü*„—ÑÂ0Ë % _timestamp‚1.7842861115227394e+09 _runtime‚13303.648370714 _step‚1287 ' lr-AdamW/pg1‚5.5180800977748514e-05 ' lr-AdamW/pg2‚5.5180800977748514e-05 trainer/global_step‚15649‡ ’ÞkR4Ã0/ _wandbruntime‚13303  _runtime‚13303e† WjÄ0€P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1288 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124150634 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144992871 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124150636 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytÒÁœ#es.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16559878100992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144982894 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27413408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2379611054088192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2383899767249810 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚13303.64982533 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16560219440128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269143507 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16534620326802 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16534960876692 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2396170915149824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16560219440128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144982894 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2383900124839060 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144992871 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16741660216320 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61112890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2367365163962368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2379611037048832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27413411 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523904621733888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33699748 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523904603121664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2396171273528320à | (Å0…P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16560219440128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269143507 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16534620326802 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16534960876692 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2396170915149824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16560219440128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144982894 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2383900124839060 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144992871 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16741660216320 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61112890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2367365163962368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2379611037048832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27413411 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523904621733888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33699748 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523904603121664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2396171273528320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1288 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124150634 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144992871 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124150636 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16559878100992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144982894 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27413408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2379611054088192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2383899767249810 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚13303.64982533ˆ 4üÏñ4Æ0/ _wandbruntime‚13304  _runtime‚133042]¯ 'Ç0ŠN T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2384047288975404 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269160205 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚15649 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16560890727936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269150212 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784286112134708e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16560890727936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124158266 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269160205 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2396319527192064 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2379758278078464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33701549 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16535971930526 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16535631380524 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124158268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16742767580672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚540680710891008 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16561232074240 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2396319527192064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16561232074240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2384047646564766 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145001937 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2379758295117824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _runtime‚13304.260514933 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144991946 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2379758295117824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27415156 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚540680818073088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523938031880192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2367511657594880 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145001937 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27415153 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124158268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33701808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _step‚1289 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269150212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2396319168806400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61116702 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124158266 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523938031880192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2367511657594880 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145001937 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMoüš×#nitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27415153 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124158268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33701808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _step‚1289 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269150212 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2396319168806400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61116702 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124158266 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269160205 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚15649 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16560890727936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269150212 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784286112134708e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16560890727936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124158266 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269160205 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2396319527192064 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2379758278078464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33701549 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16535971930526 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16535631380524 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124158268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16742767580672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚540680710891008 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888‰ Ó¯¦ 4É0/ _wandbruntime‚13304  _runtime‚13304<˜Ê0ˆ #loss/total_loss‚6.8045903479162 epoch‚39 % _timestamp‚1.7842861121387174e+09 0loss/FRt1/assignment_loss‚1.5143202744572368 3loss/SRqqt1/assignment_loss‚0.34099665810405194 trainer/global_step‚15649 _step‚1290 0loss/FRt2/assignment_loss‚0.6908417506146919 2loss/FBt1/assignment_loss‚0.012699705401464955 3loss/FBt2/assignment_loss‚6.089396558053295e-05 /loss/FRt2/detection_loss‚1.1907077576950245 1loss/SRqqt2/detection_loss‚0.5964723765677712 /loss/FBt2/detection_loss‚0.4445417384034954 _runtime‚13304.264059436 3loss/SRqqt2/assignment_loss‚0.48264197424700794 /loss/FRt1/detection_loss‚0.4702056628653218 1loss/SRqqt1/detection_loss‚0.7473801369426631 /loss/FBt1/detection_loss‚0.3137214186518896å}:®Ë0 trainer/global_step‚15649 _step‚1290 0loss/FRt2/assignment_loss‚0.6908417506146919 2loss/FBt1/assignment_loss‚0.012699705401464955 3loss/FBt2/assignment_loss‚6.089396558053295e-05 /loss/FRt2/detection_loss‚1.1907077576950245 1loss/SRqqt2/detection_loss‚0.5964723765677712 /loss/FBt2/detection_loss‚0.4445417384034954 _runtime‚13304.264059436 3loss/SRqqt2/assignment_loss‚0.48264197424700794 /loss/FRt1/detection_loss‚0.4702056628653218 1loss/SRqqt1/detection_loss‚0.7473801369426631 /loss/FBt1/detection_loss‚0.3137214186518896 #loss/total_loss‚6.8045903479162 epoch‚39 % _timestamp‚1.7842861121387174e+09 0loss/FRt1/assignment_loss‚1.5143202744572368 3loss/SRqqt1/assignment_loss‚0.34099665810405194Š Ã~ë&žÌ0‚(:“  —èÒÄ¥£ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.258' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.251599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h¸,Í0‚(:ý  —èҌݒ¥$ disk./.usageGB‚548.7983818054199 disk.dm-1.in‚0# disk.dm-0.out‚54271.67041015625& disk.nvme0n1.out‚54271.67041015625 disk.sda.in‚0 disk.sda3.out‚ 534.0859375& proc.memory.availableMB‚ 428.140625 network.recv‚ 3.53251e+06 disk.sda2.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚54271.67041015625) proc.memory.percent‚17.22586154937744) disk./.usagePercent‚36.83031987547466 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 network.sent‚ 2.4922468e+07 disk.dm-1.out‚ 534.0859375 disk.dm-0.in‚144927.01953125 disk.sda1.in‚0 disk.sda2.in‚0! disk.rbd1.out‚117.32470703125 disk.sda1.out‚0$ memory_percent‚97.38683700561523 proc.cpu.threads‚17 disk.sda.out‚ 534.0859375# disk.nvme0n1.in‚144927.01953125 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144927.01953125# proc.memory.rssMB‚ 2822.28515625 cpu‚26.019784090283462ÀÙ{<“Î0‚(:ˆ ¯—èҪ䉤 gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÆJîÏ0‚(:ú ¯—èÒ˜ŽÄ¥ cpu‚26.019442362236365 disk.sda1.out‚0( disk.nvme0c0n1.out‚54276.60693359375 disk.rbd1.in‚0.0625) proc.memory.percent‚17.22586154937744 network.sent‚ 2.4985476e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚548.7984848022461 disk.dm-0.in‚144927.01953125 disk.sda2.out‚0 disk.dm-1.out‚ 534.40625 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144927.01953125$ memory_percent‚97.38852977752686& disk.nvme0n1.out‚54276.60693359375) disk./.usagePercent‚36.83032678767808# disk.dm-0.out‚54276.60693359375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.in‚0! disk.rbd1.out‚117.32470703125# proc.memory.rssMB‚ 2822.28515625 disk.sda1.in‚0 network.recv‚ 3.536853e+06 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚144927.01953125 disk.sda.out‚ 534.40625 disk.sda3.in‚0( proc.memory.availableMB‚ 427.86328125 disk.sda.in‚0 disk.sda3.out‚ 534.40625Ó–ê’Ð0‚(:‡ ¾—èÒ‘¢ gpu.0.gpu‚29.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã8MuôÑ0‚(:é ¾—èÒØ׳¥ disk.dm-1.out‚ 534.65625( disk./.usagePercent‚36.8303395880548 disk.drbd2165.out‚0( proc.memory.availableMB‚ 428.12890625 disk.sda2.out‚0 disk.sda3.out‚ 534.65625 disk.rbd1.in‚0.0625 disk.rbd3.out‚0% disk.nvme0c0n1.out‚54283.77734375 network.sent‚ 2.4992628e+07 disk.sda1.in‚0 disk.sda.in‚0 network.recv‚ 3.540112e+06 disk.rbd0.out‚0 disk.sda3.in‚0! disk.rbd1.out‚117.38720703125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2822.28515625# disk.nvme0n1.out‚54283.77734375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 144927.734375! disk.nvme0n1.in‚ 144927.734375 disk.dm-1.in‚0$ memory_percent‚97.38690853118896 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 144927.734375 disk.dm-0.out‚54283.77734375$ disk./.usageGB‚548.7986755371094 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 534.65625) proc.memory.percent‚17.22586154937744 cpu‚26.01910350015445þzlµ4Ò0/ _wandbruntime‚13343  _runtime‚13343ý²þtÉÓ0à & lr-AdamW/pg2‚5.397022292074046e-05 trainer/global_step‚15699 % _timestamp‚1.7842861512912297e+09 _runtime‚13343.41683072 _step‚1291 & lr-AdamW/pg1‚5.397022292074046e-05Ì $ÎÔ0È % _timestamp‚1.7842861512912297e+09 _runtime‚13343.41683072 _step‚1291 & lr-AdamW/pg1‚5.397022292074046e-05 & lr-AdamW/pg2‚5.397022292074046e-05 trainer/global_step‚15699‹ ĪG®4Õ0/ _wandbruntime‚13343  _runtime‚13343Lî[³(Ö0€P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2374690680516608 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2403584020511744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16793066966528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2403583662133248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16610914941952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542369198426624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16793138766336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2386973105569792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33803874 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27500661 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2374690697555968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145437346 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842861512924604e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2386973105569792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚13343.418367181 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124532236 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16585577363600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124532234 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16610573602816 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2386973088530432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2386973088530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71799808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438°bO$×0…P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124532234 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16610573602816 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2386973088530432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2386973088530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71799808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2374690680516608 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2403584020511744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16793066966528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2403583662133248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16610914941952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542369198426624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16793138766336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2386973105569792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33803874 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27500661 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2374690697555968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145437346 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.põÍ8ȼeak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842861512924604e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2386973105569792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚13343.418367181 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124532236 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16585577363600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888Œ ¿ ¡ê4Ø0/ _wandbruntime‚13344  _runtime‚13344ìõS'Ù0‹N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269996257 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124539866 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2391423796645274 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61308370 % _timestamp‚1.7842861519045494e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2403732274175488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16794168604672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269986264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚13344.030420747 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145446398 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16611586229760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33805695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16586247867432 trainer/global_step‚15699 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61308098 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2387120329560064 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16794084229120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16611586229760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124539866 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2387120346599424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542403554035712 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2391423439055912 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269996257 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124539868 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27502403 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2403731915789824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145456389 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚542403657023488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61308098 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2387120329560064 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16794084229120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16611586229760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124539866 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2387120346599424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542403554035712 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2391423439055912 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269996257 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124539868 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27502403 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2403731915789824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145456389 K“o4Û0/ _wandbruntime‚13344  _runtime‚13344â”–„ Ü0‡ 0loss/FBt1/detection_loss‚0.30636525891313793 /loss/FBt2/detection_loss‚0.5128648564646677 epoch‚39 % _timestamp‚1.7842861519071112e+09 %loss/total_loss‚7.023096479416832 _step‚1294 0loss/FRt1/assignment_loss‚1.4695507564179884 2loss/FBt2/assignment_loss‚0.003941039681959627 /loss/FRt2/detection_loss‚1.2110257078545446 1loss/SRqqt1/detection_loss‚0.8374058148405372 0loss/SRqqt2/detection_loss‚0.643837532558223 trainer/global_step‚15699 3loss/SRqqt1/assignment_loss‚0.31005524275394497 2loss/SRqqt2/assignment_loss‚0.4447808083150134 2loss/FBt1/assignment_loss‚0.016366946790732126 _runtime‚13344.03245103 0loss/FRt2/assignment_loss‚0.7544356249945259 /loss/FRt1/detection_loss‚0.5124668898315576KÌ~®Ý0Œ _runtime‚13344.03245103 0loss/FRt2/assignment_loss‚0.7544356249945259 /loss/FRt1/detection_loss‚0.5124668898315576 0loss/FBt1/detection_loss‚0.30636525891313793 /loss/FBt2/detection_loss‚0.5128648564646677 epoch‚39 % _timestamp‚1.7842861519071112e+09 %loss/total_loss‚7.023096479416832 _step‚1294 0loss/FRt1/assignment_loss‚1.4695507564179884 2loss/FBt2/assignment_loss‚0.003941039681959627 /loss/FRt2/detection_loss‚1.2110257078545446 1loss/SRqqt1/detection_loss‚0.8374058148405372 0loss/SRqqt2/detection_loss‚0.643837532558223 trainer/global_step‚15699 3loss/SRqqt1/assignment_loss‚0.31005524275394497 2loss/SRqqt2/assignment_loss‚0.4447808083150134 2loss/FBt1/assignment_loss‚0.016366946790732126Ž š H“Þ0‚(:ˆ Í—èқ߅¤ gpu.0.gpu‚35.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êý@—÷ß0‚(:ì Í—èÒ¶Þ¿¥ disk.dm-1.out‚ 534.85546875$ memory_percent‚97.38833904266357) disk./.usagePercent‚36.83034854831849! disk.nvme0c0n1.in‚ 144928.5625 disk.sda2.in‚0) proc.memory.percent‚17.22586154937744 disk.sda.in‚0! disk.rbd1.out‚117.38720703125 disk.rbd3.out‚0 cpu‚26.01876446010566 disk.drbd2165.out‚0 disk.dm-0.in‚ 144928.5625# disk.nvme0n1.out‚54294.49609375# proc.memory.rssMB‚ 2822.28515625 network.sent‚ 2.499555e+07 disk.rbd0.out‚0% disk.nvme0c0n1.out‚54294.49609375 disk.rbd3.in‚0 network.recv‚ 3.543612e+06 disk.rbd0.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 427.89453125$ disk./.usageGB‚548.7988090515137 disk.sda.out‚ 534.85546875 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚ 144928.5625 disk.sda1.out‚0 disk.sda3.out‚ 534.85546875 disk.dm-0.out‚54294.49609375 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0KºVžà0‚(:“ Ü—èÒŸú·¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.717200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gzˆá0‚(:û Ü—èÒçü¥ disk.rbd1.in‚0.0625! disk.rbd1.out‚117.44970703125 disk.sda.in‚0' proc.memory.availableMB‚ 427.9609375) proc.memory.percent‚17.22586154937744) disk./.usagePercent‚36.83064782112602 disk.dm-1.out‚ 537.66015625 cpu‚26.018391162529884 disk.rbd3.in‚0# disk.nvme0n1.in‚144929.27734375 disk.sda.out‚ 537.66015625# proc.memory.rssMB‚ 2822.28515625 disk.dm-0.in‚144929.27734375$ disk.nvme0n1.out‚54314.353515625 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144929.27734375 network.recv‚ 3.547721e+06$ disk./.usageGB‚548.8032684326172 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 537.66015625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚54314.353515625 disk.sda2.in‚0# memory_percent‚97.3879337310791! disk.dm-0.out‚54314.353515625 disk.drbd2165.in‚0 network.sent‚ 2.5058538e+07 disk.sda1.out‚0 disk.sda3.in‚0\=*®žâ0‚(:“ ë—èÒ¨žý£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.762800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[(¢Kã0‚(:ý ë—èÒ¼Œå¤ disk.rbd3.in‚0( proc.memory.availableMB‚ 427.96484375* disk./.usagePercent‚36.830407174043835 disk.rbd0.out‚0 disk.sda.out‚ 541.1796875" disk.dm-0.out‚54334.7802734375 proc.cpu.threads‚17 network.sent‚ 2.5063685e+07% disk.nvme0c0n1.in‚144930.10546875 cpu‚26.01805497422093 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚144930.10546875' disk.nvme0c0n1.out‚54334.7802734375$ disk./.usageGB‚548.7996826171875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 541.1796875! disk.rbd1.out‚117.44970703125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0.0625% disk.nvme0n1.out‚54334.7802734375# proc.memory.rssMB‚ 2822.28515625 disk.sda.in‚0$ memory_percent‚97.38790988922119) proc.memory.percent‚17.22586154937744# disk.nvme0n1.in‚144930.10546875 network.recv‚ 3.551286e+06 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 541.1796875V¡24ä0/ _wandbruntime‚13383  _runtime‚13383”ë¤ÆËå0Å _step‚1295 ' lr-AdamW/pg1‚5.2771357755229724e-05 ' lr-AdamW/pg2‚5.2771357755229724e-05 trainer/global_step‚15749 $ _timestamp‚1.784286191189354e+09 _runtime‚13383.314976792 DïZÐæ0Ê $ _timestamp‚1.784286191189354e+09 _runtime‚13383.314976792 _step‚1295 ' lr-AdamW/pg1‚5.2771357755229724e-05 ' lr-AdamW/pg2‚5.2771357755229724e-05 trainer/global_step‚15749 ¢³–4ç0/ _wandbruntime‚13383  _runtime‚13383M´2ƒAè0P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16661610443776 trainer/global_step‚15749 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33907710 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527247497586688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2382016231149568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16661269104640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2398652425000076 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16636193850508 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2410996767495168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16661269104640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚13383.316443461 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2394335157051392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16661610443776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270815611 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61495883 NFDeviceStatsMonitor.onÒ?þûÆ_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7842861911903994e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544091868799488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16844476304896 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2398652067410826 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124913836 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2410996767495168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145891798 _step‚1296 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2410996409116672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2394335157051392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27587911 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124913836 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145901775 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61495618 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2410996409116672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2394335140012032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33907972 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27587908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚544091973891584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124913834 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16635853300618˜+»T (é0†P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27587908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚544091973891584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124913834 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16635853300618 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16661610443776 trainer/global_step‚15749 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33907710 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527247497586688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2382016231149568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16661269104640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2398652425000076 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16636193850508 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2410996767495168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16661269104640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚13383.316443461 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2394335157051392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16661610443776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270815611 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61495883 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7842861911903994e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544091868799488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16844476304896 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2398652067410826 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124913836 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2410996767495168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145891798 _step‚1296 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2410996409116672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2394335157051392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27587911 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124913836 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145901775 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61495618 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2410996409116672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2394335140012032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33907972 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 "e%4ê0/ _wandbruntime‚13383  _runtime‚13383J–'1'ë0ŠN SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚527280907732992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61499438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124921466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16636864354340 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270822316 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2382162741821440 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270832309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16662281731584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124921468 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124921468 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2394482398081024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16662281731584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33910038 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270832309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2398799946725782 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270822316 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 trainer/global_step‚15749 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16845415336448 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16637204904342 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2411145021158912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61499694 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544126428154368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1297 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2411145021158912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145910841 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16662623077888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145910841 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2394482398081024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16662623077888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145900850 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7842861918023407e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2382162724782080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2411144662773248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16845501809152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0a?eGÇì0N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 trainer/global_step‚15749 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16845415336448 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16637204904342 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2411145021158912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61499694 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544126428154368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1297 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2411145021158912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145910841 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16662623077888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145910841 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2394482398081024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16662623077888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145900850 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7842861918023407e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2382162724782080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2411144662773248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16845501809152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚527280907732992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61499438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124921466 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16636864354340 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270822316 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2382162741821440 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270832309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16662281731584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124921468 4.DeviceStatsMoniˆ‡ûN tor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124921468 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2394482398081024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16662281731584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33910038 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270832309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2398799946725782 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270822316 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F4í0/ _wandbruntime‚13383  _runtime‚13383'¬Œcî0ˆ 0loss/FRt1/detection_loss‚0.48848541441521676 1loss/SRqqt2/detection_loss‚0.6197280107513273 %loss/total_loss‚6.509956807233044 /loss/FRt2/assignment_loss‚0.767928819335502 2loss/FBt1/assignment_loss‚0.003062117272729136 3loss/FBt2/assignment_loss‚0.0004950994106031209 /loss/FRt2/detection_loss‚1.1065315709560808 /loss/FBt1/detection_loss‚0.3124332975299741 trainer/global_step‚15749 _step‚1298 0loss/FRt1/assignment_loss‚1.4535956733094322 1loss/SRqqt2/assignment_loss‚0.408722887609947 1loss/SRqqt1/detection_loss‚0.7864072867588878 0loss/FBt2/detection_loss‚0.27214217017014586 epoch‚39 % _timestamp‚1.7842861918061726e+09 2loss/SRqqt1/assignment_loss‚0.2904244597131983 _runtime‚13383.931513421ÿüÔÑï0 0loss/FRt1/detection_loss‚0.48848541441521676 1loss/SRqqt2/detection_loss‚0.6197280107513273 %loss/total_loss‚6.509956807233044 /loss/FRt2/assignment_loss‚0.767928819335502 2loss/FBt1/assignment_loss‚0.003062117272729136 3loss/FBt2/assignment_loss‚0.0004950994106031209 /loss/FRt2/detection_loss‚1.1065315709560808 /loss/FBt1/detection_loss‚0.3124332975299741 trainer/global_step‚15749 _step‚1298 0loss/FRt1/assignment_loss‚1.4535956733094322 1loss/SRqqt2/assignment_loss‚0.408722887609947 1loss/SRqqt1/detection_loss‚0.7864072867588878 0loss/FBt2/detection_loss‚0.27214217017014586 epoch‚39 % _timestamp‚1.7842861918061726e+09 2loss/SRqqt1/assignment_loss‚0.2904244597131983 _runtime‚13383.931513421’ úù§žð0‚(:’ ú—èÒ´þ…£ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚246.98' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.395999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.gl³þñ0‚(:ó ú—èÒŠóµ¦ disk.rbd3.in‚0# disk.nvme0n1.in‚144930.10546875 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0) disk./.usagePercent‚36.83031091521096 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚117.51220703125 disk.rbd3.out‚0# proc.memory.rssMB‚ 2822.28515625 disk.dm-1.out‚ 543.40625! disk.dm-0.out‚54338.806640625 disk.drbd2165.out‚0 network.sent‚ 2.5068703e+07$ disk./.usageGB‚548.7982482910156$ disk.nvme0n1.out‚54338.806640625 disk.sda.in‚0 disk.sda.out‚ 543.40625 disk.sda1.out‚0% disk.nvme0c0n1.in‚144930.10546875 disk.rbd1.in‚0.0625 disk.sda3.out‚ 543.40625 cpu‚26.01773062238103 network.recv‚ 3.554404e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚144930.10546875$ memory_percent‚97.38681316375732( proc.memory.availableMB‚ 428.14453125) proc.memory.percent‚17.22586154937744& disk.nvme0c0n1.out‚54338.806640625òmk‘ò0‚(:† ‰˜èÒè÷ù¢ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01cRwýó0‚(:ò ‰˜èÒâè…¥" disk.nvme0n1.in‚144930.8203125& disk.nvme0n1.out‚54350.81982421875 disk.sda1.out‚0 disk.drbd2165.out‚0! disk.rbd1.out‚117.51220703125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚26.017400547961692 disk.rbd0.in‚0( disk.nvme0c0n1.out‚54350.81982421875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚543.9375 disk.dm-0.in‚144930.8203125$ memory_percent‚97.39089012145996 proc.cpu.threads‚17 disk.rbd3.in‚0 network.recv‚ 3.55905e+06 disk.drbd2165.in‚0 disk.sda.out‚543.9375 disk.dm-1.out‚543.9375# disk./.usageGB‚548.798641204834 disk.sda1.in‚0$ disk.nvme0c0n1.in‚144930.8203125 disk.sda3.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 427.4765625 network.sent‚ 2.5131724e+07* disk./.usagePercent‚36.830337283986985# proc.memory.rssMB‚ 2822.28515625 disk.rbd1.in‚0.0625# disk.dm-0.out‚54350.81982421875) proc.memory.percent‚17.22586154937744Œ)Ì4ô0/ _wandbruntime‚13423  _runtime‚13423HO¤ÏÌõ0Æ ' lr-AdamW/pg2‚5.1584282482085716e-05 trainer/global_step‚15799 % _timestamp‚1.7842862311208944e+09 _runtime‚13423.246495906 _step‚1299 ' lr-AdamW/pg1‚5.1584282482085716e-05{lù¼Ñö0Ë ' lr-AdamW/pg1‚5.1584282482085716e-05 ' lr-AdamW/pg2‚5.1584282482085716e-05 trainer/global_step‚15799 % _timestamp‚1.7842862311208944e+09 _runtime‚13423.246495906 _step‚1299“ x;þ4÷0/ _wandbruntime‚13423  _runtime‚13423Ë Á+(ø0€P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16712305945600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146346250 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61687077 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27675158 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146346250 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16711964606464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2389341764743168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528918935513088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16895618357760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2418409514478592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2389341747703808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61686795 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545814652933632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2418409156100096 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125295436 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16686469787526 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16712305945600 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125295436 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16711964606464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146356227 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2406028575080584 trainer/global_step‚15799 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2401697208532992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16895717420544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7842862311220076e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2418409514478592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚528918916900864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2401697191493632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125295434 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34011637 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271651663 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2418409156100096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27675161 _runtime‚13423.24776926 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2401697208532992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2406028217491334 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16712305945600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146346250 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61687077 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27675158 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146346250 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16711964606464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2389341764743168 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528918935513088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16895618357760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2418409514478592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2389341747703808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61686795 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545814652933632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2418409156100096 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125295436 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16686469787526 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16712305945600 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125295436 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16711964606464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146356227 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2406028575080584 trainer/global_step‚15799 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2401697208532992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16895717420544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 % _timestamp‚1.7842862311220076e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2418409514478592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚528918916900864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2401697191493632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125295434 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34011637 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271651663 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2418409156100096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27675161 _runtime‚13423.24776926 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2401697208532992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2406028217491334 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125303068 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271658368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61690612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34013979 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271658368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2401844449562624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚545849097989120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2418557768142336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16687821391250 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚1301 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271668361 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528952364271616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61690885 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚528952345659392 _runtime‚13423.863810632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545848974029824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_traiùÓq:˜n_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2401844432523264 trainer/global_step‚15799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842862317379677e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125303066 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16896733717504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27676903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16896628370432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2406175739216928 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2418557768142336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2401844449562624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16713318579712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27676906 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2389488258375680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146355302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16687480841248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16712977233408 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146365293 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146355302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2389488275415040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146365293 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16712977233408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271668361 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146365293 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16712977233408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271668361 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125303068 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271658368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61690612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34013979 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271658368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2401844449562624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚545849097989120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2418557768142336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16687821391250 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚1301 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271668361 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528952364271616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61690885 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚528952345659392 _runtime‚13423.863810632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545848974029824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2401844432523264 trainer/global_step‚15799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7842862317379677e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125303066 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16896733717504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27676903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16896628370432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2406175739216928 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2418557768142336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2401844449562624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16713318579712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27676906 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2389488258375680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146355302 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16687480841248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16712977233408 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146365293 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146355302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2389488275415040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0• ÂY°54ý0/ _wandbruntime‚13423  _runtime‚13423õ=@þ0Š 3loss/SRqqt2/assignment_loss‚0.45479559261969915 % _timestamp‚1.7842862317399402e+09 0loss/FRt1/assignment_loss‚1.4793497755898255 1loss/SRqqt1/detection_loss‚0.8475228149435973 1loss/SRqqt2/detection_loss‚0.5720333256016509 trainer/global_step‚15799 2loss/SRqqt1/assignment_loss‚0.4773781413213139 3loss/FBt2/assignment_loss‚6.666155727401702e-05 0loss/FRt1/detection_loss‚0.45001126188556595 /loss/FRt2/detection_loss‚1.1461438177218188 /loss/FBt1/detection_loss‚0.4226877869610165 epoch‚39 1loss/FBt1/assignment_loss‚0.00417990869178633 /loss/FBt2/detection_loss‚0.6050057963188737 &loss/total_loss‚7.1593660427767505 _runtime‚13423.865278393 _step‚1302 0loss/FRt2/assignment_loss‚0.7001911595643285 ¬UDÿ0 _step‚1302 0loss/FRt2/assignment_loss‚0.7001911595643285 3loss/SRqqt2/assignment_loss‚0.45479559261969915 % _timestamp‚1.7842862317399402e+09 0loss/FRt1/assignment_loss‚1.4793497755898255 1loss/SRqqt1/detection_loss‚0.8475228149435973 1loss/SRqqt2/detection_loss‚0.5720333256016509 trainer/global_step‚15799 2loss/SRqqt1/assignment_loss‚0.4773781413213139 3loss/FBt2/assignment_loss‚6.666155727401702e-05 0loss/FRt1/detection_loss‚0.45001126188556595 /loss/FRt2/detection_loss‚1.1461438177218188 /loss/FBt1/detection_loss‚0.4226877869610165 epoch‚39 1loss/FBt1/assignment_loss‚0.00417990869178633 /loss/FBt2/detection_loss‚0.6050057963188737 &loss/total_loss‚7.1593660427767505 _runtime‚13423.865278393– Û‚_‘€1‚(:† ˜˜èÒÌá’£ gpu.0.gpu‚50.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ ”i1‚(:ö ˜˜èÒÁ™æ¤$ disk./.usageGB‚548.7986106872559 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 544.3046875 disk.dm-1.in‚0! disk.rbd1.out‚117.57470703125( proc.memory.availableMB‚ 427.49609375 cpu‚26.01708524993866! disk.dm-0.out‚54358.939453125$ disk.nvme0c0n1.in‚144931.6484375 disk.sda3.out‚ 544.3046875 disk.rbd3.in‚0$ memory_percent‚97.39077091217041# proc.memory.rssMB‚ 2822.28515625 disk.dm-1.out‚ 544.3046875 disk.rbd3.out‚0$ disk.nvme0n1.out‚54358.939453125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0) proc.memory.percent‚17.22586154937744 disk.dm-0.in‚144931.6484375" disk.nvme0n1.in‚144931.6484375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚54358.939453125 disk.sda2.in‚0 disk.rbd1.in‚0.0625 network.sent‚ 2.5136784e+07 network.recv‚ 3.562299e+06) disk./.usagePercent‚36.83033523592671 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0ôuR‘‚1‚(:† §˜èÒú¢ gpu.0.gpu‚52.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É\¬Qƒ1‚(:ü §˜èÒô›è¥ disk.sda.in‚0# disk.nvme0n1.in‚144932.36328125 disk.drbd2165.out‚0 network.recv‚ 3.566942e+06 disk.sda3.in‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.22586154937744 disk.rbd3.out‚0$ disk.nvme0n1.out‚54369.880859375 proc.cpu.threads‚17* disk./.usagePercent‚36.830333443873975 disk.rbd3.in‚0 disk.dm-0.in‚144932.36328125# proc.memory.rssMB‚ 2822.28515625 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚54369.880859375 cpu‚26.016747008864606# disk./.usageGB‚548.798583984375 disk.dm-1.in‚0 disk.dm-1.out‚ 544.62109375! disk.dm-0.out‚54369.880859375' proc.memory.availableMB‚ 427.2734375 disk.sda.out‚ 544.62109375 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144932.36328125 network.sent‚ 2.5200024e+07 disk.sda3.out‚ 544.62109375! disk.rbd1.out‚117.57470703125$ memory_percent‚97.39212989807129S¥/¿‘„1‚(:† ¶˜èÒƒ¾££ gpu.0.gpu‚31.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è/oM…1‚(:ü ¶˜èÒƒî¥ disk.rbd3.out‚0# disk.nvme0n1.in‚144933.19140625& disk.nvme0n1.out‚54399.56689453125 disk.drbd2165.in‚0 disk.rbd0.out‚0 network.recv‚ 3.570098e+06* disk./.usagePercent‚36.621394174841434% disk.nvme0c0n1.in‚144933.19140625 disk.sda.out‚ 545.578125 disk.rbd0.in‚0& proc.memory.availableMB‚ 427.515625 proc.cpu.threads‚17 network.sent‚ 2.5205135e+07 disk.sda.in‚0# disk.dm-0.out‚54399.56689453125 cpu‚26.016379294047965$ disk./.usageGB‚545.6852378845215 disk.sda3.in‚0 disk.sda3.out‚ 545.578125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 disk.dm-1.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚54399.56689453125! disk.rbd1.out‚117.63720703125 disk.dm-1.out‚ 545.578125 disk.dm-0.in‚144933.19140625 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚97.39065170288086 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625÷1-j4†1/ _wandbruntime‚13462  _runtime‚13462+dÞÖʇ1Ä trainer/global_step‚15849 % _timestamp‚1.7842862708165245e+09 _runtime‚13462.942120961 _step‚1303 & lr-AdamW/pg1‚5.040907334493486e-05 & lr-AdamW/pg2‚5.040907334493486e-05꽬Ãψ1É _step‚1303 & lr-AdamW/pg1‚5.040907334493486e-05 & lr-AdamW/pg2‚5.040907334493486e-05 trainer/global_step‚15849 % _timestamp‚1.7842862708165245e+09 _runtime‚13462.942120961— bD–4‰1/ _wandbruntime‚13462  _runtime‚1346206n³(Š1P XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125677036 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2413404725161092 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125677034 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61876275 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚530590373439488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2425822261462016 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2425821903083520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2409059260014592 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16763001447424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚547537119954944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125677036 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16946765127680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27762408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16737086274434 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16737426824324 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2425822261462016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚530590354827264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚13462.943466852 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2409059242975232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146800702 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842862708176882e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2409059242975232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16946851607552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27762411 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2425821903083520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34114128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚15849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125677034 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0°­ÒÙ·‹1†P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146810679 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2413404367571842 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batßjŒDU&ch_start/inactive_split.all.current‚264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547537225047040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2396667281297408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146800702 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34113867 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16762660108288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2409059260014592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16762660108288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16763001447424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2396667298336768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2425821903083520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34114128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚15849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125677034 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125677036 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2413404725161092 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125677034 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61876275 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚530590373439488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2425822261462016 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2425821903083520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2409059260014592 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16763001447424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚547537119954944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125677036 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16946765127680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27762408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16737086274434 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16737426824324 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2425822261462016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚530590354827264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚13462.943466852 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2409059242975232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146800702 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842862708176882e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2409059242975232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16946851607552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27762411 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146809754 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚547571681315328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272494420 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146819745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2425970515125760 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2409206501044224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _step‚1305 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16763672735232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16764014081536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16947879117312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146819745 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16947794741760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2409206484004864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2409206484004864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34116197 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2396813791969280 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146809754 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125684668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2413551889297436 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2409206501044224 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16764014081536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272504413 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27764153 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚530623802198016 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2396813809008640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125684666 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚547571578327552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2413552246886798 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125684668 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272504413 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61880080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16947794741760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2409206484004864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2409206484004864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34116197 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2396813791969280 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146809754 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125684668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2413551889297436 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2409206501044224 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16764014081536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272504413 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27764153 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚530623802198016 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2396813809008640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125684666 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚547571578327552 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2413552246886798 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125684668 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272504413 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61880080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146809754 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚547571681315328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272494420 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146819745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888™ ±òŒ¯41/ _wandbruntime‚13463  _runtime‚13463õ³¤1ˆ /loss/FBt1/detection_loss‚0.5036750254021674 %loss/total_loss‚7.572974225440941 1loss/FBt2/assignment_loss‚0.03392848682270255 1loss/SRqqt2/detection_loss‚0.5760623595609887 _step‚1306 epoch‚39 2loss/SRqqt1/assignment_loss‚0.3060317015192184 /loss/FBt2/detection_loss‚0.9398640937038378 trainer/global_step‚15849 _runtime‚13463.563397495 0loss/FRt1/assignment_loss‚1.4789728070034447 /loss/FRt2/detection_loss‚1.1968287846624077 1loss/SRqqt1/detection_loss‚0.8305621940655238 % _timestamp‚1.7842862714378605e+09 0loss/FRt2/assignment_loss‚0.7702807372399918 2loss/SRqqt2/assignment_loss‚0.4446658086697132 4loss/FBt1/assignment_loss‚0.00042516426259901096 /loss/FRt1/detection_loss‚0.4916770625283458ùUDÇ‘1 _step‚1306 epoch‚39 2loss/SRqqt1/assignment_loss‚0.3060317015192184 /loss/FBt2/detection_loss‚0.9398640937038378 trainer/global_step‚15849 _runtime‚13463.563397495 0loss/FRt1/assignment_loss‚1.4789728070034447 /loss/FRt2/detection_loss‚1.1968287846624077 1loss/SRqqt1/detection_loss‚0.8305621940655238 % _timestamp‚1.7842862714378605e+09 0loss/FRt2/assignment_loss‚0.7702807372399918 2loss/SRqqt2/assignment_loss‚0.4446658086697132 4loss/FBt1/assignment_loss‚0.00042516426259901096 /loss/FRt1/detection_loss‚0.4916770625283458 /loss/FBt1/detection_loss‚0.5036750254021674 %loss/total_loss‚7.572974225440941 1loss/FBt2/assignment_loss‚0.03392848682270255 1loss/SRqqt2/detection_loss‚0.5760623595609887š 漣’1‚(:‡ ŘèÒæ¿‹¥ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚242.965' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.593 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðƒð*“1‚(:ÿ ŘèÒô¬¥¦! disk.rbd1.out‚117.63720703125( proc.memory.availableMB‚ 427.72265625 proc.cpu.threads‚17 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚144933.2265625& disk.nvme0n1.out‚54432.75341796875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625# disk.dm-0.out‚54432.75341796875$ memory_percent‚97.38938808441162) proc.memory.percent‚17.22586154937744# disk./.usageGB‚545.689640045166" disk.nvme0n1.in‚144933.2265625 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 548.18359375 disk.rbd3.out‚0# proc.memory.rssMB‚ 2822.28515625) disk./.usagePercent‚36.62168960753595 disk.rbd0.in‚0 disk.rbd3.in‚0 cpu‚26.016078987976133 disk.dm-0.in‚144933.2265625( disk.nvme0Û¤#àc0n1.out‚54432.75341796875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 2.5223072e+07 network.recv‚ 3.573468e+06 disk.sda.out‚ 548.18359375 disk.sda3.out‚ 548.18359375ßùt“”1‚(:ˆ Ô˜èÒñ Ð¢ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Î$#ö•1‚(:ë Ô˜èÒóÓÔ¥ disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚552.375 disk.rbd3.out‚0! disk.dm-0.out‚54450.861328125 network.recv‚ 3.577726e+06 disk.sda.out‚552.375 disk.dm-1.in‚0 disk.rbd1.in‚0.0625 disk.sda3.in‚0 disk.dm-0.in‚144933.94140625 disk.rbd0.in‚0$ memory_percent‚97.39031791687012 cpu‚26.0157311645696% disk.nvme0c0n1.in‚144933.94140625$ disk.nvme0n1.out‚54450.861328125 disk.drbd2165.out‚0 network.sent‚ 2.5273085e+07 disk.dm-1.out‚552.375 disk.sda1.out‚0 proc.cpu.threads‚17# disk./.usageGB‚545.688346862793! disk.rbd1.out‚117.69970703125 disk.drbd2165.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚54450.861328125 disk.rbd3.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 427.5703125# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744* disk./.usagePercent‚36.621602820981835 disk.sda.in‚0# disk.nvme0n1.in‚144933.94140625÷#aÞ’–1‚(:‡ ã˜èÒÓ¥¸£ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.535' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.307 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê<ðgú—1‚(:ï ã˜èÒÑ„¤ network.recv‚ 3.5811e+06( disk./.usagePercent‚36.6219136141284 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚144934.76953125& disk.nvme0c0n1.out‚54472.982421875 disk.sda1.out‚0! disk.rbd1.out‚117.69970703125 disk.drbd2165.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 427.328125 disk.rbd0.out‚0# disk.nvme0n1.in‚144934.76953125 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.39179611206055 network.sent‚ 2.5278203e+07 disk.dm-1.out‚ 554.78125! disk.dm-0.out‚54472.982421875 cpu‚26.015424591100025% disk.nvme0c0n1.in‚144934.76953125$ disk./.usageGB‚545.6929779052734 disk.rbd0.in‚0 disk.sda.out‚ 554.78125 disk.sda2.in‚0) proc.memory.percent‚17.22586154937744$ disk.nvme0n1.out‚54472.982421875 disk.sda3.out‚ 554.78125 disk.rbd3.out‚0# proc.memory.rssMB‚ 2822.28515625í”÷4˜1/ _wandbruntime‚13502  _runtime‚13502 €Ê™1Ä & lr-AdamW/pg2‚4.924580582526408e-05 trainer/global_step‚15899 % _timestamp‚1.7842863107704144e+09 _runtime‚13502.897394242 _step‚1307 & lr-AdamW/pg1‚4.924580582526408e-05hìôÏš1É _runtime‚13502.897394242 _step‚1307 & lr-AdamW/pg1‚4.924580582526408e-05 & lr-AdamW/pg2‚4.924580582526408e-05 trainer/global_step‚15899 % _timestamp‚1.7842863107704144e+09› iÿ¦r4›1/ _wandbruntime‚13502  _runtime‚13502¯Î=#(œ1P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2416421311496192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16787702761342 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273313788 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126058636 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16813696949248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126058636 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16998019024384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16813696949248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2420780875241600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2420780517652350 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16788043311232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7842863107712007e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147255154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚15899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34217586 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚549259830390272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2403992831930368 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273323767 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1308 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126058634 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2416421294456832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2416421294456832 _runtime‚13502.897474503 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2416421311496192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27849658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27849661 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16813355610112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34217824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62067485 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532261792753664 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2433234650066944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147255154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532261811365888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚169979262530567 Å (1†P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147255154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚15899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34217586 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚549259830390272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2403992831930368 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273323767 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1308 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126058634 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2416421294456832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2416421294456832 _runtime‚13502.897474503 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2416421311496192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27849658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27849661 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16813355610112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34217824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62067485 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532261792753664 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2433234650066944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147255154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532261811365888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16997926253056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2416421311496192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16787702761342 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273313788 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126058636 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16813696949248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126058636 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16998019024384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16813696949248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2420780875241600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2420780517652350 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16788043311232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7842863107712007e+09œ <}\Î4ž1/ _wandbruntime‚13503  _runtime‚13503ç—*h!Ÿ1ŒN VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16814368237056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27851406 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27851403 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚549294160129024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2416568535486464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16788713815064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273340465 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62070997 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16998938616832 trainer/global_step‚15899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147274197 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273330472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _step‚1309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273340465 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62071266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2416568552525824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚549294279894016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147264206 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34219860 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16814709583360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2433383262109184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2416568552525824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2433383262109184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34219594 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147274197 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bzö¸ªytes.all.freed‚2420928039377944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2416568535486464 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2433383262109184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34219594 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147274197 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2420928039377944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2416568535486464 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273340465 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62070997 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16998938616832 trainer/global_step‚15899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147274197 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273330472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _step‚1309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273340465 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62071266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2416568552525824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚549294279894016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147264206 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34219860 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16814709583360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 Ö,) 4¡1/ _wandbruntime‚13503  _runtime‚13503¿wOt¢1‰ 1loss/SRqqt1/detection_loss‚0.7967340838194421 1loss/SRqqt2/detection_loss‚0.5746344999168973 %loss/total_loss‚6.833575018471051 _step‚1310 /loss/FRt1/assignment_loss‚1.446322025071345 _runtime‚13503.510145793 epoch‚39 trainer/global_step‚15899 0loss/FRt2/assignment_loss‚0.7857248871176092 3loss/SRqqt1/assignment_loss‚0.28786363079905664 2loss/FBt1/assignment_loss‚0.032501206802875535 0loss/FRt1/detection_loss‚0.49020693816772215 /loss/FRt2/detection_loss‚1.1900078065179736 /loss/FBt1/detection_loss‚0.4433595266039153 /loss/FBt2/detection_loss‚0.3145510673020908 % _timestamp‚1.7842863113848183e+09 2loss/SRqqt2/assignment_loss‚0.4681921024212359 3loss/FBt2/assignment_loss‚0.0034772439308878344 £›£1Ž epoch‚39 trainer/global_step‚15899 0loss/FRt2/assignment_loss‚0.7857248871176092 3loss/SRqqt1/assignment_loss‚0.28786363079905664 2loss/FBt1/assignment_loss‚0.032501206802875535 0loss/FRt1/detection_loss‚0.49020693816772215 /loss/FRt2/detection_loss‚1.1900078065179736 /loss/FBt1/detection_loss‚0.4433595266039153 /loss/FBt2/detection_loss‚0.3145510673020908 % _timestamp‚1.7842863113848183e+09 2loss/SRqqt2/assignment_loss‚0.4681921024212359 3loss/FBt2/assignment_loss‚0.0034772439308878344 1loss/SRqqt1/detection_loss‚0.7967340838194421 1loss/SRqqt2/detection_loss‚0.5746344999168973 %loss/total_loss‚6.833575018471051 _step‚1310 /loss/FRt1/assignment_loss‚1.446322025071345 _runtime‚13503.510145793ž \hž¤1‚(:“ ò˜èÒÂøÓ¢ gpu.0.gpu‚79.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚135.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.181599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'•KÚó¥1‚(:è ò˜èÒ¶–¡¤ disk.sda.out‚ 556.234375 disk.dm-0.in‚ 144935.484375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0# disk./.usageGB‚545.758358001709$ disk.nvme0c0n1.out‚ 54558.9453125 disk.drbd2165.out‚0 cpu‚26.015093557500233 network.sent‚ 2.5283193e+07 disk.sda3.out‚ 556.234375 disk.rbd3.in‚0! disk.rbd1.out‚117.76220703125 disk.rbd3.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 427.453125# disk.nvme0c0n1.in‚ 144935.484375 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625 network.recv‚ 3.584551e+06 disk.dm-1.out‚ 556.234375 disk.dm-0.out‚ 54558.9453125! disk.nvme0n1.in‚ 144935.484375" disk.nvme0n1.out‚ 54558.9453125$ memory_percent‚97.39103317260742 disk.sda2.out‚0 disk.sda3.in‚0) disk./.usagePercent‚36.62630132725775¯Ò]ß“¦1‚(:ˆ ™èÒ¬ ´¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£²-ü§1‚(:ñ ™èҫΤ disk.sda.out‚ 556.859375 disk.rbd3.out‚0% disk.nvme0n1.out‚54567.3212890625$ memory_percent‚97.39561080932617$ disk./.usageGB‚545.7591743469238 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 556.859375 disk.rbd3.in‚0 disk.dm-0.in‚ 144935.484375" disk.dm-0.out‚54567.3212890625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-1.out‚ 556.859375 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744) disk./.usagePercent‚36.62635611287008 disk.rbd0.out‚0 disk.sda.in‚0! disk.rbd1.out‚117.76220703125 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 426.703125 cpu‚26.014743974627056 network.recv‚ 3.58932e+06 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 144935.484375' disk.nvme0c0n1.out‚54567.3212890625 disk.sda3.in‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 144935.484375 network.sent‚ 2.5346195e+07”¹Eñ4¨1/ _wandbruntime‚13542  _runtime‚13542ö4½(Ê©1Ä trainer/global_step‚15949 % _timestamp‚1.7842863507431688e+09 _runtime‚13542.868977454 _step‚1311 & lr-AdamW/pg1‚4.809455463757245e-05 & lr-AdamW/pg2‚4.809455463757245e-058Ê@¹Ïª1É trainer/global_step‚15949 % _timestamp‚1.7842863507431688e+09 _runtime‚13542.868977454 _step‚1311 & lr-AdamW/pg1‚4.809455463757245e-05 & lr-AdamW/pg2‚4.809455463757245e-05Ÿ `«wt4«1/ _wandbruntime‚13542  _runtime‚13542xÀ¥y(¬1‚P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2423783345938432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126440234 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147719583 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2423783345938432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34320766 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚550982277451264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2428157025322108 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126440236 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147719583 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842863507444787e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2440647755428864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17048926999040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚15949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚550982157679104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62257400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147709606 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147709606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17049028158976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126440236 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16838319248250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16864392451072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27936908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2440647397050368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16864051111936 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2423783362977792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62257677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2440647755428864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2440647397050368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16838659798140 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚550982157679104 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62257400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147709606 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147709606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17049028158976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126440236 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16838319248250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16864392451072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27936908 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2440647397050368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16864051111936 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2423783362977792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62257677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2440647755428864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2440647397050368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_trainÄDé!·_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16838659798140 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2423783345938432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34320766 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚550982277451264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2428157025322108 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126440236 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147719583 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7842863507444787e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2440647755428864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17048926999040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169  5)È4®1/ _wandbruntime‚13543  _runtime‚13543`+]V'¯1‹N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2440796009092608 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17049953227776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17050041797632 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16865405085184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147728649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27938653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147718658 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147718658 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2440795650706944 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2423930604007424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34322571 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126447866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27938656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2440796009092608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274166524 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2428304189458452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126447866 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16865405085184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚15949 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 % _timestamp‚1.7842863513563159e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62261224 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126447868 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1313 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551016612666368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274166524 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2411464876195840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16839670851974 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16865063738880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126447868 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2423930586968064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16839330301972 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274176517 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2440796009092608 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17049953227776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17050041797632 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16865405085184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147728649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27938653 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147718658 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147718658 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2440795650706944 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2423930604007424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34322571 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126447866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27938656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2440796009092608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274166524 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2428304189458452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126447866 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16865405085184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚15949 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 % _timestamp‚1.7842863513563159e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62261224 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126447868 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1313 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551016612666368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274166524 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2411464876195840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16839670851974 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16865063738880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126447868 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2423930586968064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16839330301972 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274176517 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 FsÁù4½1/ _wandbruntime‚13582  _runtime‚13582â¹,ŽA¾1P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148174035 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2448060502412288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34424382 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148174035 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28024158 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126821836 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552704785545728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2448060502412288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17100003458560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚13582.953366891 VADeviceStatsMonitor.on_train_batch_start/allocated_byte;þÎ’Æ s.all.freed‚2448060144033792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62448277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16914746613760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148164058 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2431145397420032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535604668606464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16914746613760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1316 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126821836 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16915087952896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274995871 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62448543 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16889276285048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126821834 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2431145414459392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16915087952896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2418643882078208 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274985892 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2435533175402616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2448060144033792 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148164058 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2431145397420032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535604668606464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16914746613760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1316 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126821836 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16915087952896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274995871 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62448543 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16889276285048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126821834 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2431145414459392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16915087952896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2418643882078208 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274985892 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2435533175402616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2448060144033792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28024158 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126821836 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552704785545728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2448060502412288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17100003458560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚13582.953366891 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2448060144033792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62448277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16914746613760 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275012569 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126829468 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535638097364992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16915759240704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2435680697128322 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚15999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2448208756076032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535638115977216 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148173110 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34426419 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62452045 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148183101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28025903 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2431292655489024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2431292638449664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2435680339538960 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2418790409789440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126829468 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126829466 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Õ·[gÆÂ1N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28025903 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2431292655489024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2431292638449664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2435680339538960 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2418790409789440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126829468 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126829466 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552739128145920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2448208397690368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16915759240704 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148173110 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275012569 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126829468 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535638097364992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16915759240704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2435680697128322 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 trainer/global_step‚15999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2448208756076032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535638115977216 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148173110 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34426419 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62452045 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148183101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856¥ £e‡N4Ã1/ _wandbruntime‚13583  _runtime‚13583ó­8Ä1Œ %loss/total_loss‚6.972820763776097 0loss/FRt2/assignment_loss‚0.7418980989116855 3loss/SRqqt1/assignment_loss‚0.32774890309105636 4loss/FBt2/assignment_loss‚4.3723899440547854e-05 /loss/FRt1/detection_loss‚0.5014649075232122 /loss/FBt2/detection_loss‚0.2792196739736193 % _timestamp‚1.7842863914389086e+09 _step‚1318 0loss/FRt1/assignment_loss‚1.5237356231550965 2loss/FBt1/assignment_loss‚0.009135849226995675 _runtime‚13583.564245034 3loss/SRqqt2/assignment_loss‚0.48050526515617503 /loss/FRt2/detection_loss‚1.2355276498947818 1loss/SRqqt1/detection_loss‚0.8444476974691497 epoch‚39 trainer/global_step‚15999 1loss/SRqqt2/detection_loss‚0.6552215437502974 0loss/FBt1/detection_loss‚0.37387182772458727ÉÓ¡Å1‘ 0loss/FRt2/assignment_loss‚0.7418980989116855 3loss/SRqqt1/assignment_loss‚0.32774890309105636 4loss/FBt2/assignment_loss‚4.3723899440547854e-05 /loss/FRt1/detection_loss‚0.5014649075232122 /loss/FBt2/detection_loss‚0.2792196739736193 % _timestamp‚1.7842863914389086e+09 _step‚1318 0loss/FRt1/assignment_loss‚1.5237356231550965 2loss/FBt1/assignment_loss‚0.009135849226995675 _runtime‚13583.564245034 3loss/SRqqt2/assignment_loss‚0.48050526515617503 /loss/FRt2/detection_loss‚1.2355276498947818 1loss/SRqqt1/detection_loss‚0.8444476974691497 epoch‚39 trainer/global_step‚15999 1loss/SRqqt2/detection_loss‚0.6552215437502974 0loss/FBt1/detection_loss‚0.37387182772458727 %loss/total_loss‚6.972820763776097¦ ¼7[žÆ1‚(:“ ½™èÒž´­£ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ÃtþÇ1‚(:ó ½™èÒ£³¿¥ network.recv‚ 3.603218e+06 disk.sda3.in‚0$ memory_percent‚97.39429950714111 disk.sda3.out‚ 557.984375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 426.91796875) proc.memory.percent‚17.22586154937744 disk.sda1.out‚0 disk.sda.out‚ 557.984375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 557.984375 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚144938.5703125& disk.nvme0c0n1.out‚54805.361328125# proc.memory.rssMB‚ 2822.28515625 cpu‚26.01531726891627 disk.rbd1.in‚0.0625! disk.rbd1.out‚117.94970703125 disk.dm-0.in‚144938.5703125 disk.sda2.out‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚144938.5703125 disk.drbd2165.in‚0 disk.rbd3.in‚0 network.sent‚ 2.5423412e+07) disk./.usagePercent‚36.62175463344964$ disk./.usageGB‚545.6906089782715! disk.dm-0.out‚54805.361328125$ disk.nvme0n1.out‚54805.361328125¿ ÙË4È1/ _wandbruntime‚13600  _runtime‚13600Ž~̲Š=É1„{ 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6291220784187317 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22256699204444885 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2655590772628784 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24049343168735504 )loss/val_total_loss‚8.035025555809243 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037400536239147186 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20243752002716064 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6704007983207703 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5674912929534912 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20184803009033203 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.701622486114502 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27697545289993286 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6100776195526123 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6377800107002258 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0713602676987648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6743426322937012 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4624260365962982 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5089676976203918 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976046681404114 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2528029978275299 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8694403767585754 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6327039003372192 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2771787643432617 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6095530390739441 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2543596029281616 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07487990707159042 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.01893535628914833 ,jet/accuracy_4_of_4‚0.008297258988022804 _step‚1319 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49436578154563904 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7008692026138306 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2510174512863159 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26714280247688293 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7939941883087158 +particle/sensitivity‚0.6168045401573181 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8992068767547607 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2238752245903015 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6176173686981201 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6176173686981201 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5772641897201538 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046141162514686584 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8694403767585754 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2778589725494385 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30232736468315125 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6706856489181519 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960803985595703 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6694715023040771 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01937750168144703 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019494682550430298 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24433912336826324 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7763344049453735 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7970646619796753 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23979413509368896 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2104547619819641 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.624000608921051 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6343731880187988 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8028470277786255 particle/accuracy_1_of_5‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_2_of_5‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979740977287292 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7694258093833923 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.587093710899353 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6989392042160034 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6326050162315369 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952129125595093 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6557527780532837 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_2_of_2‚0.15996864438056946 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.9035592675209045 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6093183159828186 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7200988531112671 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.672313392162323 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.019611552357673645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8466014862060547 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 *jet/accuracy_1_of_1‚0.3355667293071747 'particle/f_score‚0.6263486742973328 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7268969416618347 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5331867933273315 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32386788725852966 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.383882611989975 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9906997680664062 (particle/accuracy‚0.8419789671897888 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_4_of_5‚1 *jet/accuracy_1_of_3‚0.9266067743301392 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2098839432001114 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6441901326179504 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 _runtime‚13600.69745838 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2401229292154312 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6969388127326965 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996785581111908 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997230052947998 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.044359173625707626 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0840405821800232 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920265674591064 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6validation_average_jet_accuracy‚0.3780503273010254 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 5 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.52040034532547 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6629334688186646 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01937750168144703 /particle/accuracy_4_of_4‚0.9573301672935486 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6815192699432373 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2673718333244324 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.53083336353302 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995938241481781 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7686401605606079 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03780710697174072 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7699998021125793 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2667578458786011 *jet/accuracy_2_of_3‚0.5793724060058594 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8473057746887207 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 epoch‚39 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25879374146461487 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.548843502998352 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7290298938751221 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26714280247688293 +jet/accuracy_3_of_3‚0.05971185490489006 *jet/accuracy_2_of_4‚0.8049338459968567 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36575639247894287 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6403508186340332 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954886436462402 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9073758125305176 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7008692026138306 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019068405032157898 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9930327534675598 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3660424053668976 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6674352884292603 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9926129579544067 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6927626729011536 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2166285216808319 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6096675395965576 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2359505593776703 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28067269921302795 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916567325592 trainer/global_step‚15999 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904581904411316 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.639096736907959 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.927219569683075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9764662384986877 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33607906103134155 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24220407009124756 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6159377098083496 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9852258563041687 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6415154933929443 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_3_of_3‚1 % _timestamp‚1.7842864085711768e+09¼¸*ºÊ1‰{ ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2667578458786011 *jet/accuracy_2_of_3‚0.5793724060058594 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8473057746887207 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 epoch‚39 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25879374146461487 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.548843502998352 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7290298938751221 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26714280247688293 +jet/accuracy_3_of_3‚0.05971185490489006 *jet/accuracy_2_of_4‚0.8049338459968567 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36575639247894287 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6403508186340332 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954886436462402 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9073758125305176 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7008692026138306 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019068405032157898 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9930327534675598 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3660424053668976 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6674352884292603 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9926129579544067 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6927626729011536 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2166285216808319 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6096675395965576 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2359505593776703 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28067269921302795 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916567325592 trainer/global_step‚15999 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904581904411316 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.639096736907959 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.927219569683075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9764662384986877 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33607906103134155 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24220407009124756 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6159377098083496 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9852258563041687 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6415154933929443 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_3_of_3‚1 % _timestamp‚1.7842864085711768e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6291220784187317 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22256699204444885 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2655590772628784 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24049343168735504 )loss/val_total_loss‚8.035025555809243 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037400536239147186 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20243752002716064 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6704007983207703 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5674912929534912 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20184803009033203 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.701622486114502 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27697545289993286 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6100776195526123 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6377800107002258 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0713602676987648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6743426322937012 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4624260365962982 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5089676976203918 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976046681404114 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2528029978275299 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8694403767585754 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6327039003372192 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2771787643432617 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6095530390739441 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2543596029281616 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07487990707159042 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.01893535628914833 ,jet/accuracy_4_of_4‚0.008297258988022804 _step‚1319 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49436578154563904 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7008692026138306 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2510174512863159 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26714280247688293 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7939941883087158 +particle/sensitivity‚0.6168045401573181 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.89920ÛhR‹Õ68767547607 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2238752245903015 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6176173686981201 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6176173686981201 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5772641897201538 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046141162514686584 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8694403767585754 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2778589725494385 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30232736468315125 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6706856489181519 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960803985595703 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6694715023040771 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01937750168144703 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019494682550430298 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24433912336826324 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7763344049453735 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7970646619796753 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23979413509368896 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2104547619819641 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.624000608921051 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6343731880187988 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8028470277786255 particle/accuracy_1_of_5‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_2_of_5‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979740977287292 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7694258093833923 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.587093710899353 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6989392042160034 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6326050162315369 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952129125595093 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6557527780532837 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_2_of_2‚0.15996864438056946 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.9035592675209045 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6093183159828186 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7200988531112671 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.672313392162323 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.019611552357673645 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8466014862060547 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 *jet/accuracy_1_of_1‚0.3355667293071747 'particle/f_score‚0.6263486742973328 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7268969416618347 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5331867933273315 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32386788725852966 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.383882611989975 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9906997680664062 (particle/accuracy‚0.8419789671897888 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_4_of_5‚1 *jet/accuracy_1_of_3‚0.9266067743301392 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2098839432001114 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6441901326179504 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 _runtime‚13600.69745838 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2401229292154312 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6969388127326965 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996785581111908 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997230052947998 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.044359173625707626 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0840405821800232 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920265674591064 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6validation_average_jet_accuracy‚0.3780503273010254 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 5 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.52040034532547 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6629334688186646 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01937750168144703 /particle/accuracy_4_of_4‚0.9573301672935486 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6815192699432373 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2673718333244324 5 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.53083336353302 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6432291269302368 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995938241481781 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7686401605606079 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03780710697174072 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7699998021125793 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757§ n “Ë1‚(:ˆ Ì™èÒ‡…¡£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚252.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý÷¤Ì1‚(:ü Ì™èÒœ¡Ù¦ disk.sda.in‚0# proc.memory.rssMB‚ 2822.28515625 disk.rbd1.in‚0.0625& disk.nvme0n1.out‚54808.62744140625 disk.sda.out‚ 558.3359375( proc.memory.availableMB‚ 426.90234375 proc.cpu.threads‚17$ disk./.usageGB‚545.7518844604492 disk.sda2.in‚0 disk.sda3.out‚ 558.3359375) proc.memory.percent‚17.22586154937744 disk.rbd3.out‚0 disk.drbd2165.in‚0 network.sent‚ 2.5486425e+07 disk.dm-1.in‚0# disk.dm-0.out‚54808.62744140625 disk.sda1.in‚0 disk.rbd0.in‚0 network.recv‚ 3.60708e+06 disk.dm-1.out‚ 558.3359375 disk.rbd3.in‚0 disk.dm-0.in‚144938.5703125 disk.sda2.out‚0) disk./.usagePercent‚36.62586688247214! disk.rbd1.out‚118.01220703125" disk.nvme0n1.in‚144938.5703125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚54808.62744140625 disk.sda3.in‚0$ memory_percent‚97.39439487457275 cpu‚26.018079419083513$ disk.nvme0c0n1.in‚144938.5703125D­ƒL“Í1‚(:ˆ Û™èÒƒù·¤ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@5ŸëþÎ1‚(:ó Û™èÒÎÑ…¦ network.sent‚ 2.554601e+07$ disk.nvme0n1.out‚54882.306640625 disk.dm-1.out‚ 558.640625 disk.rbd1.in‚0.0625! disk.rbd1.out‚118.01220703125! disk.dm-0.out‚54882.306640625 disk.sda1.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚36.62575347113447$ disk./.usageGB‚545.7501945495605 disk.sda.in‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚144939.3984375$ memory_percent‚97.39158153533936) proc.memory.percent‚17.22586154937744 network.recv‚ 3.611411e+06 disk.drbd2165.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚54882.306640625 disk.sda.out‚ 558.640625 disk.dm-0.in‚144939.3984375# proc.memory.rssMB‚ 2822.28515625 cpu‚26.017751236516602 disk.sda1.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 427.36328125 disk.sda2.out‚0 disk.sda3.out‚ 558.640625 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚144939.3984375Sãi#›Ï1‚(: ê™èÒœ Â¢ gpu.0.gpu‚7.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w”i] Ð1‚(:‚ ê™èÒÿ˜•¥ disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144940.11328125 disk.sda.out‚ 561.46484375' proc.memory.availableMB‚ 426.8671875# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17$ disk./.usageGB‚545.6873588562012 disk.dm-1.in‚0 disk.dm-1.out‚ 561.46484375 disk.sda2.in‚0$ memory_percent‚97.39460945129395 network.sent‚ 2.5549038e+07# disk.dm-0.out‚54961.76025390625 disk.rbd0.in‚0 disk.sda3.out‚ 561.46484375 disk.dm-0.in‚144940.11328125 disk.sda1.in‚0 disk.sda.in‚0) proc.memory.percent‚17.22586154937744 cpu‚26.017424573299866 disk.rbd3.in‚0# disk.nvme0n1.in‚144940.11328125 disk.sda3.in‚0& disk.nvme0n1.out‚54961.76025390625 disk.drbd2165.out‚0 network.recv‚ 3.614788e+06) disk./.usagePercent‚36.62153651503047! disk.rbd1.out‚118.01220703125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚54961.76025390625 disk.sda2.out‚0 disk.rbd3.out‚0+œË4Ñ1/ _wandbruntime‚13640  _runtime‚13640,i <ÊÒ1Ä & lr-AdamW/pg1‚4.582839625244061e-05 & lr-AdamW/pg2‚4.582839625244061e-05 trainer/global_step‚16049 % _timestamp‚1.7842864483678117e+09 _runtime‚13640.494426502 _step‚1320Oƒ ÏÓ1É trainer/global_step‚16049 % _timestamp‚1.7842864483678117e+09 _runtime‚13640.494426502 _step‚1320 & lr-AdamW/pg1‚4.582839625244061e-05 & lr-AdamW/pg2‚4.582839625244061e-05¨ S»Ÿ4Ô1/ _wandbruntime‚13640  _runtime‚13640ß¾c(Õ1P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2457420549251584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2444835708422433 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚1321 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148698809 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554461397412864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127324100 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2457420190873088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2440437032591360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127324102 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842864483686013e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16983516660224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2440437032591360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276022911 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62674263 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34560605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127324100 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148698809 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28113928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2457420190873088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28113931 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148688832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚537292018632704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16983516660224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17169473648640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2457420549251584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2444835350833183 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17169378780160 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2440437015552000 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2440437015552000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2427878095925248 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16957271947295 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0‘€Ôõ%Ö1†P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2427878078885888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276012932 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127324102 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16983175321088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554461510893568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2457420190873088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2440437032591360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127324102 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842864483686013e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16983516660224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2440437032591360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276022911 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62674263 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34560605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127324100 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148698809 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28113928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2457420190873088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28113931 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148688832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚537292018632704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16983516660224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17169473648640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2457420549251584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2444835350833183 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17169378780160 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2440437015552000 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2440437015552000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2427878095925248 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16957271947295 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2457420549251584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2444835708422433 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚1321 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148698809 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554461397412864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127324100 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276039609 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127331734 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554495965656064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2440584256581632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚13641.105550674 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148707875 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2428024606597120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127331732 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28115673 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16984529294336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34562664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2444982872558777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚537325447391232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1322 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62678065 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148697884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842864489797635e+09ƒ^ M'Ù1N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2440584256581632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚13641.105550674 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148707875 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2428024606597120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127331732 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28115673 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16984529294336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34562664 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2444982872558777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚537325447391232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1322 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62678065 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148697884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842864489797635e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148697884 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127331732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚16049 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276039609 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62678340 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17170423665664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2440584256581632 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16984187948032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2457568802915328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276039609 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127331734 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554495965656064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591ª / 1A4Ú1/ _wandbruntime‚13641  _runtime‚13641u¡¡LÛ1‰ % _timestamp‚1.7842864489830818e+09 2loss/SRqqt1/assignment_loss‚0.4079204635740112 2loss/FBt1/assignment_loss‚0.003319814183383299 /loss/FRt1/detection_loss‚0.4988253456246391 1loss/SRqqt1/detection_loss‚0.8008104486010816 _runtime‚13641.108422435 _step‚1323 3loss/FBt2/assignment_loss‚4.107619981587826e-05 &loss/total_loss‚7.0061452103163315 trainer/global_step‚16049 0loss/FRt2/assignment_loss‚0.8309240836136754 /loss/FRt2/detection_loss‚1.1870211532277544 epoch‚40 0loss/FRt1/assignment_loss‚1.4279027997770093 2loss/SRqqt2/assignment_loss‚0.4393723476393852 1loss/SRqqt2/detection_loss‚0.6275188803494796 /loss/FBt1/detection_loss‚0.3105017580016654 /loss/FBt2/detection_loss‚0.4719870395244315æ £Ü1Ž 1loss/SRqqt1/detection_loss‚0.8008104486010816 _runtime‚13641.108422435 _step‚1323 3loss/FBt2/assignment_loss‚4.107619981587826e-05 &loss/total_loss‚7.0061452103163315 trainer/global_step‚16049 0loss/FRt2/assignment_loss‚0.8309240836136754 /loss/FRt2/detection_loss‚1.1870211532277544 epoch‚40 0loss/FRt1/assignment_loss‚1.4279027997770093 2loss/SRqqt2/assignment_loss‚0.4393723476393852 1loss/SRqqt2/detection_loss‚0.6275188803494796 /loss/FBt1/detection_loss‚0.3105017580016654 /loss/FBt2/detection_loss‚0.4719870395244315 % _timestamp‚1.7842864489830818e+09 2loss/SRqqt1/assignment_loss‚0.4079204635740112 2loss/FBt1/assignment_loss‚0.003319814183383299 /loss/FRt1/detection_loss‚0.4988253456246391« këåêÝ1‚(:’ ù™èÒƒÝ㢠gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚245.093' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚49.01859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«’éðÞ1‚(:ü ù™èҊ홤 disk.sda1.out‚0% disk.nvme0c0n1.in‚144940.94140625 disk.dm-1.out‚ 563.84765625 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚54979.064453125 disk.sda.out‚ 563.84765625 disk.sda3.in‚0 disk.dm-1.in‚0 network.sent‚ 2.5553977e+07! disk.rbd1.out‚118.07470703125 disk.sda2.in‚0$ memory_percent‚97.39470481872559) proc.memory.percent‚17.22586154937744 network.recv‚ 3.618363e+06 disk.dm-0.in‚144940.94140625$ disk./.usageGB‚545.6944961547852# disk.nvme0n1.in‚144940.94140625& disk.nvme0c0n1.out‚54979.064453125 disk.sda.in‚0 disk.rbd1.in‚0.0625$ disk.nvme0n1.out‚54979.064453125 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 563.84765625' proc.memory.availableMB‚ 426.8515625# proc.memory.rssMB‚ 2822.28515625) disk./.usagePercent‚36.62201550512703 disk.drbd2165.out‚0 disk.rbd0.in‚0 cpu‚26.017080726738016r^û ß1‚(:’ ˆšèҹص¢ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚267.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.589999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X¤mIûà1‚(:𠈚èÒ°ÿÒ¤# disk./.usageGB‚545.694522857666 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 144941.65625$ memory_percent‚97.39463329315186! disk.rbd1.out‚118.07470703125 disk.dm-0.in‚ 144941.65625 disk.rbd0.in‚0 disk.sda.out‚ 565.546875 disk.dm-1.in‚0 disk.dm-1.out‚ 565.546875 disk.rbd3.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚ 3.622514e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 565.546875 disk.rbd3.in‚0 network.sent‚ 2.561722e+07 disk.sda3.in‚0% disk.nvme0n1.out‚54989.3662109375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 144941.65625' disk.nvme0c0n1.out‚54989.3662109375# proc.memory.rssMB‚ 2822.28515625* disk./.usagePercent‚36.622017297179774 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 426.86328125 cpu‚26.016768537288467" disk.dm-0.out‚54989.3662109375 disk.sda.in‚0) proc.memory.percent‚17.22586154937744ŒWØ‘á1‚(:† —šèÒšÙ°¢ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Kä7â1‚(:ö —šèÒÅõˤ proc.cpu.threads‚17 disk.nvme0n1.in‚ 144941.65625 disk.sda.out‚ 566.1796875 disk.sda2.in‚0 network.sent‚ 2.5622243e+07 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0$ memory_percent‚97.39460945129395 network.recv‚ 3.625815e+06& disk.nvme0n1.out‚54995.36181640625 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.dm-0.in‚ 144941.65625# proc.memory.rssMB‚ 2822.28515625 disk.rbd3.out‚0# disk.dm-0.out‚54995.36181640625$ disk./.usageGB‚545.6947059631348 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0) disk./.usagePercent‚36.62202958554141 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 144941.65625 disk.dm-1.in‚0 disk.dm-1.out‚ 566.1796875 cpu‚26.016440775151718 disk.sda3.out‚ 566.1796875! disk.rbd1.out‚118.13720703125( disk.nvme0c0n1.out‚54995.36181640625' proc.memory.availableMB‚ 426.8671875) proc.memory.percent‚17.22586154937744h2Ûó4ã1/ _wandbruntime‚13680  _runtime‚13680†B_MÊä1Ä & lr-AdamW/pg1‚4.471363460611895e-05 & lr-AdamW/pg2‚4.471363460611895e-05 trainer/global_step‚16099 % _timestamp‚1.7842864883232195e+09 _runtime‚13680.448771321 _step‚1324@><Ïå1É & lr-AdamW/pg1‚4.471363460611895e-05 & lr-AdamW/pg2‚4.471363460611895e-05 trainer/global_step‚16099 % _timestamp‚1.7842864883232195e+09 _runtime‚13680.448771321 _step‚1324¬ 5ü§4æ1/ _wandbruntime‚13680  _runtime‚13680 >#ìç1P P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2447799084072960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28201181 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17008228984093 trainer/global_step‚16099 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2447799067033600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149143284 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2464832937856512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17034212162048 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17034212162048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34664890 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149153261 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276858963 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62866071 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1325 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2464833296235008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/seg‚% ³ment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17033870822912 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2447799067033600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149153261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276848984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34664625 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚538963475171328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17220958737920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2452211858502941 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚556184415297024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127705700 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2452211500913691 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17221041023488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17007888434203 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127705700 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚538963456559104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127705702 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2464833296235008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7842864883242087e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508SKT (è1†P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2464832937856512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149143284 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2435203612479488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2464832937856512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17034212162048 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17034212162048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34664890 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149153261 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276858963 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62866071 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1325 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2464833296235008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17033870822912 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2447799067033600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149153261 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276848984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34664625 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚538963475171328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17220958737920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2452211858502941 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚556184415297024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127705700 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2452211500913691 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17221041023488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17007888434203 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127705700 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚538963456559104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127705702 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2464833296235008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7842864883242087e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2447799084072960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28201181 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17008228984093 trainer/global_step‚16099 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2447799067033600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149143284 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276865668 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149152336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62869563 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚16099 _step‚1326 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2435350140190720 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2464981191513088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2452359380228647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚13681.061514379 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784286488935601e+09 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276875661 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538996903929856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17222073736192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17035224796160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34666907 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149162327 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2447946325102592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127713332 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34666640 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168Éësë1ŽN K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62869563 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚16099 _step‚1326 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2435350140190720 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2464981191513088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2452359380228647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚13681.061514379 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784286488935601e+09 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276875661 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538996903929856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚172220731‹B¹¡736192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17035224796160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34666907 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149162327 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2447946325102592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127713332 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34666640 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚94599168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17034883449856 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17034883449856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276865668 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276865668 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149152336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154® ‡Òoç4ì1/ _wandbruntime‚13681  _runtime‚13681â+í1ˆ 2loss/FBt2/assignment_loss‚6.96110121714355e-05 1loss/SRqqt1/detection_loss‚0.8073466464362553 $loss/total_loss‚6.96380384995504 epoch‚40 _runtime‚13681.063579738 _step‚1327 0loss/FRt2/assignment_loss‚0.7086238353548193 2loss/FBt1/assignment_loss‚0.006807995894170704 0loss/FRt1/detection_loss‚0.49622565724606793 $ _timestamp‚1.784286488938251e+09 3loss/SRqqt2/assignment_loss‚0.41235525734551515 1loss/SRqqt2/detection_loss‚0.6579207413817768 0loss/FBt1/detection_loss‚0.29947752916829173 trainer/global_step‚16099 0loss/FRt1/assignment_loss‚1.5491158557400124 /loss/FRt2/detection_loss‚1.2408552878934738 /loss/FBt2/detection_loss‚0.4284638652158901 2loss/SRqqt1/assignment_loss‚0.3565415672665957zI˜î1 0loss/FRt1/assignment_loss‚1.5491158557400124 /loss/FRt2/detection_loss‚1.2408552878934738 /loss/FBt2/detection_loss‚0.4284638652158901 2loss/SRqqt1/assignment_loss‚0.3565415672665957 2loss/FBt2/assignment_loss‚6.96110121714355e-05 1loss/SRqqt1/detection_loss‚0.8073466464362553 $loss/total_loss‚6.96380384995504 epoch‚40 _runtime‚13681.063579738 _step‚1327 0loss/FRt2/assignment_loss‚0.7086238353548193 2loss/FBt1/assignment_loss‚0.006807995894170704 0loss/FRt1/detection_loss‚0.49622565724606793 $ _timestamp‚1.784286488938251e+09 3loss/SRqqt2/assignment_loss‚0.41235525734551515 1loss/SRqqt2/detection_loss‚0.6579207413817768 0loss/FBt1/detection_loss‚0.29947752916829173 trainer/global_step‚16099¯ cøï“ï1‚(:ˆ ¦šèÒèƒü¢ gpu.0.gpu‚80.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FgJøð1‚(:í ¦šèÒÍ“¥ disk.drbd2165.out‚0) proc.memory.percent‚17.22586154937744 network.recv‚ 3.629026e+06 disk.rbd1.in‚0.0625 disk.rbd3.out‚0' disk.nvme0c0n1.out‚55013.3603515625) disk./.usagePercent‚36.62214120482634$ disk./.usageGB‚545.6963691711426 disk.rbd3.in‚0" disk.dm-0.out‚55013.3603515625! disk.nvme0n1.in‚ 144942.484375# proc.memory.rssMB‚ 2822.28515625 cpu‚26.016111867578967 network.sent‚ 2.5627295e+07 disk.sda1.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 144942.484375# memory_percent‚97.3954439163208 disk.dm-1.out‚568.1875! disk.rbd1.out‚118.13720703125 disk.dm-0.in‚ 144942.484375 disk.sda1.out‚0 disk.sda.out‚568.1875( proc.memory.availableMB‚ 426.73046875 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda3.out‚568.1875 disk.dm-1.in‚0% disk.nvme0n1.out‚55013.3603515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0­áMÜ“ñ1‚(:ˆ µšèÒ¦Ó¥¢ gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pðçò1‚(:ø µšèÒŠñ¤ disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 569.7109375 disk.rbd1.in‚0.0625! disk.dm-0.out‚55027.783203125$ memory_percent‚97.39542007446289# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744& disk.nvme0c0n1.out‚55027.783203125! disk.rbd1.out‚118.26220703125 disk.dm-0.in‚144943.19921875 cpu‚26.015749492081376) disk./.usagePercent‚36.62189313352566 disk.sda.in‚0 disk.sda2.out‚0$ disk./.usageGB‚545.6926727294922# disk.nvme0n1.in‚144943.19921875$ disk.nvme0n1.out‚55027.783203125 disk.sda1.out‚0 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144943.19921875 disk.rbd3.out‚0 network.sent‚ 2.5690317e+07 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 426.734375 disk.sda1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.633559e+06 disk.rbd3.in‚0 disk.sda3.out‚ 569.7109375 disk.dm-1.out‚ 569.7109375z´b4ó1/ _wandbruntime‚13720  _runtime‚13720\¼.Ìô1Æ ' lr-AdamW/pg1‚4.3611180384664663e-05 ' lr-AdamW/pg2‚4.3611180384664663e-05 trainer/global_step‚16149 % _timestamp‚1.7842865283827686e+09 _runtime‚13720.508365039 _step‚1328ñ¥Ñõ1Ë ' lr-AdamW/pg1‚4.3611180384664663e-05 ' lr-AdamW/pg2‚4.3611180384664663e-05 trainer/global_step‚16149 % _timestamp‚1.7842865283827686e+09 _runtime‚13720.508365039 _step‚1328° '…EE4ö1/ _wandbruntime‚13720  _runtime‚13720تÆ(÷1€P SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2472245684839936 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2459588008583449 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17058845471001 % _timestamp‚1.7842865283837798e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277685036 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128087300 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557907166428672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28288431 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2459587650994199 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2442529146073088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63057222 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149597736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17084907663872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17084907663872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28288428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2442529163112448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17058504921111 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2472246043218432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128087302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17272356325888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557907269423616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540634894485504 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 277695015 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17272271943168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2455161118515200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540634913097728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚1329 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17084566324736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2455161135554560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2472245684839936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149607713 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128087300 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17084566324736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2472245684839936 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2459588008583449 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17058845471001 % _timestamp‚1.7842865283837798e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277685036 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128087300 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557907166428672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28288431 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2459587650994199 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2442529146073088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63057222 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149597736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17084907663872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17084907663872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28288428 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2442529163112448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17058504921111 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2472246043218432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128087302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17272356325888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557907269423616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540634894485504 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 277695015 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17272271943168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2455161118515200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540634913097728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚1329 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17084566324736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2455161135554560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2472245684839936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149607713 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128087300 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17084566324736 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17085920297984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1330 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2442675673784320 _runtime‚13721.122019201 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128094934 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277711713 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28290176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2455308376584192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17085920297984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17085578951680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149606788 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149616779 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2472394296882176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2472393938496512 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚16149 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149606788 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540668323244032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277701720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2455308359544832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540668341856256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17273286445056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63061285 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.784286528996244e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128094932 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2472394296882176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17085578951680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2459735530309155 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128094932 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34771109 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2442675656744960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63061014 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17059856524835 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 277711713 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128094934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28290173 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17085920297984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1330 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2442675673784320 _runtime‚13721.122019201 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128094934 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277711713 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28290176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2455308376584192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17085920297984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17085578951680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149606788 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149616779 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2472394296882176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2472393938496512 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚16149 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149606788 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540668323244032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277701720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2455308359544832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540668341856256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17273286445056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63061285 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.784286528996244e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128094932 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2472394296882176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17085578951680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2459735530309155 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128094932 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34771109 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2442675656744960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63061014 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17059856524835 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 277711713 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128094934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28290173 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EMÞý1Œ /loss/FRt2/detection_loss‚1.2289847251994848 epoch‚40 % _timestamp‚1.7842865289993231e+09 3loss/SRqqt2/assignment_loss‚0.41572986413381335 3loss/FBt1/assignment_loss‚0.0017454916103621728 /loss/FRt1/detection_loss‚0.5249366026487824 1loss/SRqqt1/detection_loss‚0.9727121228394522 1loss/SRqqt2/detection_loss‚0.5533211730161504 trainer/global_step‚16149 2loss/SRqqt1/assignment_loss‚0.3232620041437054 0loss/FBt1/detection_loss‚0.30680094457119605 /loss/FBt2/detection_loss‚0.3889689413875973 _step‚1331 %loss/total_loss‚6.836522376523513 _runtime‚13721.124650738 0loss/FRt1/assignment_loss‚1.4312610715560998 0loss/FRt2/assignment_loss‚0.6884819026551725 4loss/FBt2/assignment_loss‚0.00031753276169674483Yc—þ1‘ 0loss/FRt1/assignment_loss‚1.4312610715560998 0loss/FRt2/assignment_loss‚0.6884819026551725 4loss/FBt2/assignment_loss‚0.00031753276169674483 /loss/FRt2/detection_loss‚1.2289847251994848 epoch‚40 % _timestamp‚1.7842865289993231e+09 3loss/SRqqt2/assignment_loss‚0.41572986413381335 3loss/FBt1/assignment_loss‚0.0017454916103621728 /loss/FRt1/detection_loss‚0.5249366026487824 1loss/SRqqt1/detection_loss‚0.9727121228394522 1loss/SRqqt2/detection_loss‚0.5533211730161504 trainer/global_step‚16149 2loss/SRqqt1/assignment_loss‚0.3232620041437054 0loss/FBt1/detection_loss‚0.30680094457119605 /loss/FBt2/detection_loss‚0.3889689413875973 _step‚1331 %loss/total_loss‚6.836522376523513 _runtime‚13721.124650738³ tY3Óÿ1‚(:ˆ ÄšèÒ¡’ž£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚270.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-…^—ý€2‚(:ò ÄšèÒ÷”Û¦ disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0! disk.rbd1.out‚118.26220703125 disk.dm-0.out‚ 55040.5546875 disk.sda1.in‚0 disk.sda3.out‚ 571.7734375 disk.rbd3.out‚0 disk.rbd0.out‚0) proc.memory.percent‚17.22586154937744 disk.rbd3.in‚0" disk.nvme0n1.out‚ 55040.5546875% disk.nvme0c0n1.in‚144944.02734375 disk.dm-0.in‚144944.02734375 proc.cpu.threads‚17$ disk./.usageGB‚545.6926422119141 disk.sda.in‚0 disk.drbd2165.out‚0 cpu‚26.01543467997621) disk./.usagePercent‚36.62189108546539 disk.sda2.in‚0 disk.rbd1.in‚0.0625# disk.nvme0n1.in‚144944.02734375 disk.sda1.out‚0$ memory_percent‚97.39527702331543# proc.memory.rssMB‚ 2822.28515625 network.sent‚ 2.5708455e+07 disk.dm-1.out‚ 571.7734375 network.recv‚ 3.636745e+06$ disk.nvme0c0n1.out‚ 55040.5546875 disk.sda.out‚ 571.7734375 disk.sda2.out‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 426.7578125€ÌÓ5’2‚(:‡ ÓšèÒІ¤ gpu.0.gpu‚24.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚279.051' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à2>‚2‚(:ý ÓšèÒø¾ ¥ disk.drbd2165.in‚0' disk.nvme0c0n1.out‚55049.2275390625$ disk./.usageGB‚545.6936187744141 disk.sda.out‚ 572.33984375 disk.sda2.in‚0* disk./.usagePercent‚36.621956623394155 disk.dm-0.in‚144944.02734375 disk.drbd2165.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2822.28515625) proc.memory.percent‚17.22586154937744 cpu‚26.015133459754423% disk.nvme0n1.out‚55049.2275390625 network.recv‚ 3.64091e+06 disk.dm-1.out‚ 572.33984375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda3.out‚ 572.33984375$ memory_percent‚97.39179611206055& proc.memory.availableMB‚ 427.328125 disk.rbd0.in‚0% disk.nvme0c0n1.in‚144944.02734375 disk.sda3.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 2.576058e+07! disk.rbd1.out‚118.32470703125 disk.rbd3.out‚0" disk.dm-0.out‚55049.2275390625# disk.nvme0n1.in‚144944.02734375`;¶’ƒ2‚(:‡ âšèÒ‹·Á¢ gpu.0.gpu‚17.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØüDû„2‚(:ð âšèÒ˜†¥ disk.rbd1.in‚0.0625" disk.nvme0n1.in‚144944.7421875( proc.memory.availableMB‚ 427.18359375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 572.859375 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚55056.44140625 network.recv‚ 3.644166e+06$ disk./.usageGB‚545.6921310424805 disk.sda2.out‚0# disk.nvme0n1.out‚55056.44140625 disk.sda1.out‚0) proc.memory.percent‚17.22586154937744 cpu‚26.014791714961916 disk.sda2.in‚0 disk.dm-0.in‚144944.7421875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚144944.7421875 disk.sda.out‚ 572.859375( disk./.usagePercent‚36.6218567804558 disk.sda3.out‚ 572.859375 disk.rbd3.out‚0$ memory_percent‚97.39267826080322 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2822.28515625 network.sent‚ 2.5763502e+07 disk.dm-1.in‚0! disk.rbd1.out‚118.32470703125 disk.rbd3.in‚0 disk.dm-0.out‚55056.44140625 disk.sda.in‚0dèTÆ4…2/ _wandbruntime‚13760  _runtime‚13760f/f̆2Æ ' lr-AdamW/pg1‚4.2521104396652444e-05 ' lr-AdamW/pg2‚4.2521104396652444e-05 trainer/global_step‚16199 % _timestamp‚1.7842865682813072e+09 _runtime‚13760.406962795 _step‚1332U…3ºÑ‡2Ë ' lr-AdamW/pg1‚4.2521104396652444e-05 ' lr-AdamW/pg2‚4.2521104396652444e-05 trainer/global_step‚16199 % _timestamp‚1.7842865682813072e+09 _runtime‚13760.406962795 _step‚1332´ äZ1u4ˆ2/ _wandbruntime‚13760  _runtime‚13760'1g(‰2P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17135603165696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128468902 _runtime‚13760.411381847 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17135603165696 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2449854696706048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842865682826004e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17323471167488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2466963801074707 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚559629910768640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17135261826560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚16199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2462523169996800 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2462523169996800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150052188 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1333 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2479658431823360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2479658431823360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2479658790201856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2466964158663957 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2479658790201856 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2462523187036160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17135261826560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542306351024128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542306332411904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150062165 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2449854679666688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34872840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150052188 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278531067 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128468900 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2479658431823360 M=DeviceStatsMonitor.on_train_batch_start/ç<²™$active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2479658790201856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2466964158663957 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2479658790201856 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2462523187036160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17135261826560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚542306351024128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚542306332411904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150062165 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2449854679666688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34872840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150052188 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278531067 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128468900 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17135261826560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚16199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2462523169996800 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2462523169996800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150052188 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1333 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2479658431823360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17136615799808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63252629 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚13761.020518319 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63252361 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28377426 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28377423 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2450001207377920 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2450001190338560 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17110473011743 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150061240 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278547765 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17136615799808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17136274453504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128476532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150071231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2462670411026432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17324577499648 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2467111680389663 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128476534 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128476532 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17324488929792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2479807043865600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542339761170432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2479806685479936 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2462670428065792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128476534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17136274453504 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17110132461741 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842865688947012e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2462670428065792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2479807043865600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976üÇSð'2N UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2479806685479936 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2462670428065792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128476534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17136274453504 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17110132461741 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚1334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842865688947012e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2462670428065792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2479807043865600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17136615799808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63252629 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚13761.020518319 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63252361 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28377426 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28377423 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2450001207377920 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2450001190338560 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17110473011743 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150061240 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278547765 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17136615799808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17136274453504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128476532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150071231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2462670411026432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17324577499648 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2467111680389663 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128476534 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128476532 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17324488929792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2479807043865600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚542339761170432¶ EÊùL4Ž2/ _wandbruntime‚13761  _runtime‚13761…'ÙG2‰ 1loss/SRqqt1/detection_loss‚1.0012699567111791 1loss/SRqqt2/detection_loss‚0.5692317155743287 %loss/total_loss‚7.589147050470303 epoch‚40 % _timestamp‚1.7842865688968334e+09 /loss/FBt2/detection_loss‚0.7487143308935629 trainer/global_step‚16199 _step‚1335 0loss/FRt2/assignment_loss‚0.7923773118312168 3loss/FBt1/assignment_loss‚4.257115578110415e-05 /loss/FBt1/detection_loss‚0.5152952392061038 2loss/SRqqt1/assignment_loss‚0.3501801537377384 3loss/SRqqt2/assignment_loss‚0.42520066190144556 /loss/FRt2/detection_loss‚1.1295452046049272 _runtime‚13761.022162222 0loss/FRt1/assignment_loss‚1.5631071614847003 2loss/FBt2/assignment_loss‚7.36797937440551e-05 /loss/FRt1/detection_loss‚0.4941090635755756ÍbŠÞ2Ž 0loss/FRt2/assignment_loss‚0.7923773118312168 3loss/FBt1/assignment_loss‚4.257115578110415e-05 /loss/FBt1/detection_loss‚0.5152952392061038 2loss/SRqqt1/assignment_loss‚0.3501801537377384 3loss/SRqqt2/assignment_loss‚0.42520066190144556 /loss/FRt2/detection_loss‚1.1295452046049272 _runtime‚13761.022162222 0loss/FRt1/assignment_loss‚1.5631071614847003 2loss/FBt2/assignment_loss‚7.36797937440551e-05 /loss/FRt1/detection_loss‚0.4941090635755756 1loss/SRqqt1/detection_loss‚1.0012699567111791 1loss/SRqqt2/detection_loss‚0.5692317155743287 %loss/total_loss‚7.589147050470303 epoch‚40 % _timestamp‚1.7842865688968334e+09 /loss/FBt2/detection_loss‚0.7487143308935629 trainer/global_step‚16199 _step‚1335· Ý0Êùž‘2‚(:“ ñšèÒãÜé¢ gpu.0.gpu‚51.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚247.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.404199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.š™’2‚(:ú ñšèÒðçǤ disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 573.47265625 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚118.38720703125) proc.memory.percent‚17.22586154937744 network.sent‚ 2.5768561e+07$ disk./.usageGB‚545.6923942565918 disk.rbd3.in‚0# memory_percent‚97.3942756652832) disk./.usagePercent‚36.62187444497567 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 573.47265625 disk.dm-1.out‚ 573.47265625% disk.nvme0n1.out‚55065.6298828125# proc.memory.rssMB‚ 2822.28515625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0" disk.dm-0.out‚55065.6298828125 network.recv‚ 3.647493e+06$ disk.nvme0c0n1.in‚144945.5703125 disk.sda3.in‚0 disk.dm-0.in‚144945.5703125 cpu‚26.014488047014552 disk.drbd2165.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚55065.6298828125" disk.nvme0n1.in‚144945.5703125& proc.memory.availableMB‚ 426.921875xb3€Ì“2‚(:ˆ €›èÒñÛÉ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚186.8•£|ŽÇ61' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ ûG ”2‚(: €›èÒÜæܧ disk.rbd0.out‚0( disk.nvme0c0n1.out‚55073.10400390625 disk.sda2.out‚0! disk.rbd1.out‚118.38720703125# proc.memory.rssMB‚ 2822.28515625 cpu‚26.014188736793145 network.recv‚ 3.651929e+06 disk.sda.in‚0 disk.sda3.out‚ 574.57421875# disk.dm-0.out‚55073.10400390625) proc.memory.percent‚17.22586154937744 disk.dm-0.in‚144946.28515625% disk.nvme0c0n1.in‚144946.28515625# disk./.usageGB‚545.695613861084 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 574.57421875 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0$ memory_percent‚97.39432334899902 disk.sda.out‚ 574.57421875 disk.sda2.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 426.9140625 disk.rbd3.out‚0 network.sent‚ 2.5831591e+07) disk./.usagePercent‚36.62209051533456# disk.nvme0n1.in‚144946.28515625 disk.sda1.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0& disk.nvme0n1.out‚55073.10400390625 disk.rbd0.in‚0~ÔK&“•2‚(:ˆ ›èÒÝê­¢ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢ëÅÿ–2‚(:ô ›èÒåÕÙ¤ disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚144947.11328125 disk.sda1.out‚0 network.recv‚ 3.65545e+06 disk.rbd3.out‚0 disk.drbd2165.out‚0 cpu‚26.013824137231673 network.sent‚ 2.5836716e+07 disk.rbd0.out‚0 disk.rbd3.in‚0# disk.nvme0n1.in‚144947.11328125 disk.sda1.in‚0$ memory_percent‚97.39456176757812 disk.rbd0.in‚0 disk.sda3.in‚0! disk.dm-0.out‚55099.751953125) proc.memory.percent‚17.22586154937744 disk.sda.out‚ 576.8359375 disk.sda3.out‚ 576.8359375 disk.dm-1.in‚0 disk.dm-1.out‚ 576.8359375# proc.memory.availableMB‚426.875# proc.memory.rssMB‚ 2822.28515625) disk./.usagePercent‚36.62227202467634! disk.rbd1.out‚118.44970703125$ disk.nvme0n1.out‚55099.751953125 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚545.6983184814453% disk.nvme0c0n1.in‚144947.11328125& disk.nvme0c0n1.out‚55099.751953125 disk.rbd1.in‚0.0625Ki[Å4—2/ _wandbruntime‚13800  _runtime‚13800–ZûÑ̘2Æ ' lr-AdamW/pg1‚4.1443476655625956e-05 ' lr-AdamW/pg2‚4.1443476655625956e-05 trainer/global_step‚16249 % _timestamp‚1.7842866080826814e+09 _runtime‚13800.208314805 _step‚1336Õ*ßÑ™2Ë ' lr-AdamW/pg2‚4.1443476655625956e-05 trainer/global_step‚16249 % _timestamp‚1.7842866080826814e+09 _runtime‚13800.208314805 _step‚1336 ' lr-AdamW/pg1‚4.1443476655625956e-05¸ ËÛ>v4š2/ _wandbruntime‚13800  _runtime‚13800¡£ ’(›2P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17185957328384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2469885221478400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17374658626560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34976451 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150506640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63439379 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28462931 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2487071178806784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2469885238517760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279367119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543977788950528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279357140 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2487071537185280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128850500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34976706 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2457180230299648 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17160078444817 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2487071178806784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2457180213260288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1337 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17185957328384 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63439637 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17374570049536 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2469885221478400 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2469885238517760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136tGû³ (œ2†P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150516617 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17185957328384 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63439637 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17374570049536 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2469885221478400 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2469885238517760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17185957328384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2469885221478400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17374658626560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34976451 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150506640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63439379 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28462931 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2487071178806784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2469885238517760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279367119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543977788950528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279357140 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2487071537185280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128850500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34976706 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2457180230299648 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17160078444817 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2487071178806784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2457180213260288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1337 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63443133 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279383817 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279373824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150515692 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28464676 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128858134 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚544011199096832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150515692 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2470032462508032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34978460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2470032479547392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17186969955328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128858132 _runtime‚13800.820677522 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2487219432463360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2457326723932160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17161089498651 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚1338 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150525683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚16249 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2487219790849024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28464673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150525683 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.cÓV«êurrent‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63443389 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279383817 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17160748948649 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784286608694885e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚561386785933824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976ñíñï'Ÿ2N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2457326723932160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17161089498651 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚1338 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150525683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚16249 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2487219790849024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28464673 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150525683 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63443389 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279383817 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17160748948649 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784286608694885e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚561386785933824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63443133 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279383817 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279373824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150515692 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28464676 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128858134 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚544011199096832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150515692 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2470032462508032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34978460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2470032479547392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17186969955328 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128858132 _runtime‚13800.820677522 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2487219432463360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0º |»là4 2/ _wandbruntime‚13800  _runtime‚13800¯cDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280193192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129232102 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150961092 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2481716101235723 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2494483925790208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28550181 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63629845 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚16299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17236994169344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28550178 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35079667 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚545649208264704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150971069 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129232100 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2494483925790208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17236652830208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2477247289999360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129232100 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2464505746853888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35079924 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2464505763893248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1341 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 # _timestamp‚1.78428664809749e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2477247272960000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17236994169344Þ¯Ôáî¬2„P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2477247289999360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129232100 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2464505746853888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35079924 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2464505763893248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1341 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 # _timestamp‚1.78428664809749e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2477247272960000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17236994169344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2494484284168704 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17236652830208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2481716458824973 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17210354381835 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚13840.223374417 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2494484284168704 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63630105 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17425692825600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280193192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129232102 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150961092 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2481716101235723 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2494483925790208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28550181 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63629845 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚16299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17236994169344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28550178 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35079667 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚545649208264704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150971069 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129232100 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2494483925790208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17236652830208½ ø_Äk4­2/ _wandbruntime‚13840  _runtime‚13840u¶/í'®2‹N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35081930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17237665457152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2464652257525760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150980135 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280219869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17238006803456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17237665457152 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129239732 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2464652274565120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚545682637023232 % _timestamp‚1.7842866487096145e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129239734 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129239734 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28551926 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17211365435557 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2477394513989632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17238006803456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚16299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2477394513989632 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2494632179446784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63633581 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2494632537832448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2477394531028992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28551923 _step‚1342 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280219869 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2481863622961317 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280209876 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150970144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35081658 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150970144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17211705985559 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2494632537832448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚545682655635456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63633581 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2494632537832448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2477394531028992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28551923 _step‚1342 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280219869 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2481863622961317 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280209876 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150970144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35081658 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150970144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17211705985559 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2494632537832448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚545682655635456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17237665457152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2464652257525760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150980135 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280219869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17238006803456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17237665457152 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129239732 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2464652274565120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚545682637023232 % _timestamp‚1.7842866487096145e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129239734 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129239734 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28551926 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17211365435557 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2477394513989632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17238006803456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚16299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438¾ ƒìUÍ4°2/ _wandbruntime‚13840  _runtime‚13840kOîù ±2… trainer/global_step‚16299 % _timestamp‚1.7842866487124565e+09 0loss/FRt1/assignment_loss‚1.4509165494910403 2loss/SRqqt1/assignment_loss‚0.4292147477436611 2loss/FBt1/assignment_loss‚0.009849755248099449 1loss/FBt2/assignment_loss‚0.00691168356834257 1loss/SRqqt2/detection_loss‚0.6078908416404776 _step‚1343 /loss/FRt1/detection_loss‚0.5074115817664736 1loss/SRqqt1/detection_loss‚0.9562419852768025 .loss/FBt1/detection_loss‚0.465319319446951 %loss/total_loss‚7.307179791371832 _runtime‚13840.83810681 3loss/SRqqt2/assignment_loss‚0.43770675000064774 0loss/FBt2/detection_loss‚0.43783546639905263 0loss/FRt2/assignment_loss‚0.7627768010736626 .loss/FRt2/detection_loss‚1.235104309716621 epoch‚40¯yGW²2Š /loss/FRt1/detection_loss‚0.5074115817664736 1loss/SRqqt1/detection_loss‚0.9562419852768025 .loss/FBt1/detection_loss‚0.465319319446951 %loss/total_loss‚7.307179791371832 _runtime‚13840.83810681 3loss/SRqqt2/assignment_loss‚0.43770675000064774 0loss/FBt2/detection_loss‚0.43783546639905263 0loss/FRt2/assignment_loss‚0.7627768010736626 .loss/FRt2/detection_loss‚1.235104309716621 epoch‚40 trainer/global_step‚16299 % _timestamp‚1.7842866487124565e+09 0loss/FRt1/assignment_loss‚1.4509165494910403 2loss/SRqqt1/assignment_loss‚0.4292147477436611 2loss/FBt1/assignment_loss‚0.009849755248099449 1loss/FBt2/assignment_loss‚0.00691168356834257 1loss/SRqqt2/detection_loss‚0.6078908416404776 _step‚1343¿ òÌ\“‘³2‚(:† ¼›èÒœ¡ý¤ gpu.0.gpu‚30.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z|ì–ô´2‚(:é ¼›èÒŽÚ±¦) disk./.usagePercent‚36.62215246915785$ disk./.usageGB‚545.6965370178223 disk.drbd2165.in‚0 disk.dm-1.out‚ 580.71484375 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 580.71484375$ memory_percent‚96.89180850982666# proc.memory.rssMB‚ 2639.90234375* proc.memory.percent‚16.112685203552246 proc.cpu.threads‚17 disk.sda3.out‚ 580.71484375 disk.nvme0n1.in‚ 144948.65625 disk.nvme0n1.out‚ 55129.34375 disk.rbd0.out‚0( proc.memory.availableMB‚ 509.24609375 network.recv‚ 3.666232e+06 disk.dm-0.out‚ 55129.34375 disk.drbd2165.out‚0 disk.sda2.out‚0! disk.rbd1.out‚118.51220703125 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 144948.65625 disk.sda3.in‚0 cpu‚26.012818931530962 network.sent‚ 2.5909913e+07 disk.rbd1.in‚0.0625 disk.dm-1.in‚0 disk.dm-0.in‚ 144948.65625 disk.sda1.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 55129.34375 disk.sda2.in‚0 disk.sda1.in‚0$ëÛ“µ2‚(:ˆ Ë›èÒõÅØ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚192.306' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.4612 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª~–ûÿ¶2‚(:ô Ë›èÒ×–ï¤ network.sent‚ 2.5972059e+07 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 55138.765625 cpu‚26.012505341277777 network.recv‚ 3.670061e+06 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.112685203552246 disk.sda3.in‚0 disk.dm-1.out‚ 580.91015625 disk.rbd3.in‚0# disk.nvme0c0n1.out‚ 55138.765625) disk./.usagePercent‚36.62216040539141! disk.nvme0n1.out‚ 55138.765625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0n1.in‚144949.37109375 disk.dm-1.in‚0! disk.rbd1.out‚118.63720703125$ memory_percent‚96.88992500305176# proc.memory.rssMB‚ 2639.90234375 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 580.91015625 disk.sda3.out‚ 580.91015625 disk.dm-0.in‚144949.37109375 disk.rbd1.in‚0.0625% disk.nvme0c0n1.in‚144949.37109375$ disk./.usageGB‚545.6966552734375' proc.memory.availableMB‚ 509.5546875 proc.cpu.threads‚17Íw‰’·2‚(:‡ Ú›èÒû±»¢ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚204.985' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wc*è¸2‚(:ƒ Ú›èÒÆ‘´¥) disk./.usagePercent‚36.62157773224348 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 581.33203125 disk.rbd3.out‚0& disk.nvme0n1.out‚55141.23193359375$ disk./.usageGB‚545.6879730224609 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 581.33203125 disk.rbd1.in‚0.0625 disk.dm-0.in‚144949.37109375# disk.nvme0n1.in‚144949.37109375' proc.memory.availableMB‚ 509.7578125 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144949.37109375 disk.sda2.out‚0* proc.memory.percent‚16.112685203552246 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚55141.23193359375 disk.sda.in‚0 disk.rbd3.in‚0 cpu‚26.012184522886272 network.recv‚ 3.673245e+06 disk.drbd2165.out‚0 disk.sda.out‚ 581.33203125 disk.sda3.in‚0# disk.dm-0.out‚55141.23193359375# proc.memory.rssMB‚ 2639.90234375! disk.rbd1.out‚118.63720703125 network.sent‚ 2.5977051e+07$ memory_percent‚96.88868522644043 disk.dm-1.in‚0Áf~©4¹2/ _wandbruntime‚13880  _runtime‚13880°°”ʺ2Ä _runtime‚13880.075878764 _step‚1344 & lr-AdamW/pg1‚3.932584196662131e-05 & lr-AdamW/pg2‚3.932584196662131e-05 trainer/global_step‚16349 % _timestamp‚1.7842866879502327e+09¾<{Ï»2É & lr-AdamW/pg1‚3.932584196662131e-05 & lr-AdamW/pg2‚3.932584196662131e-05 trainer/global_step‚16349 % _timestamp‚1.7842866879502327e+09 _runtime‚13880.075878764 _step‚1344À œWYý4¼2/ _wandbruntime‚13880  _runtime‚13880Or0 ½2P RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151425521 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2489092608905481 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129613700 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17476759656960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17261311418633 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17287348332032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2501896672773632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1345 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2501897031152128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚16349 _runtime‚13880.077113124 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17287689671168 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547320646191104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151415544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2484609341480960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281039223 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚564797405848064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63819897 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151425521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2471831297486848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281029244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2489092251316231 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129613702 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17287689671168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚547320664803328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2484609324441600 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2501896672773632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _step‚1345 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2501897031152128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚16349 _runtime‚13880.077113124 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17287689671168 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547320646191104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151415544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2484609341480960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281039223 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚564797405848064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63819897 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151425521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2471831297486848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281029244 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2489092251316231 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129613702 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17287689671168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚547320664803328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2484609324441600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17261311418633 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17287348332032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2502044926430208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151434587 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28639173 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129621334 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2484756565471232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2489239773041825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚13880.876704398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151424596 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151424596 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17288360958976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚564831848416256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17288702305280 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2484756582510592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7842866887508893e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚16349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129621332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17262322472467 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129621334 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63823364 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2489240130631187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2471977791119360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17288360958976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17288702305280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚564831961889792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17477773466624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281045928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151434587 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129621332 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚547354093561856 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2502045284815872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281045928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63823643 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚547354074949632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2484756582510592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28639176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281055921 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281055921 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872ÚaNº‰Á2N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2484756565471232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281055921 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281055921 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17261981922465 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35184191 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2502044926430208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151434587 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28639173 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129621334 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2484756565471232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2489239773041825 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚13880.876704398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151424596 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151424596 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonYÐitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17288360958976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚564831848416256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17288702305280 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2484756582510592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7842866887508893e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚16349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129621332 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17262322472467 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129621334 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63823364 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2489240130631187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2471977791119360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17288360958976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17288702305280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚564831961889792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17477773466624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281045928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151434587 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129621332 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚547354093561856 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2502045284815872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281045928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63823643 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚547354074949632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2484756582510592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28639176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚16399 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2491971392962560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17338043833856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129995300 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2491971392962560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17527965656576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17311587355651 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64009911 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17527872885248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2509309419757056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _step‚1349 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151879973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2509309778135552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚548992084117504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281875275 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17311927905541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566519957002752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129995300 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2491971375923200 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17338043833856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2509309419757056 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚548992102729728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151869996 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151879973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35285230 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28724678 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ø| (Ð2†P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2491971392962560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17338043833856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129995300 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2491971392962560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17527965656576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17311587355651 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64009911 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17527872885248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2509309419757056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _step‚1349 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151879973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2509309778135552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚548992084117504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281875275 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17311927905541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566519957002752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129995300 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2491971375923200 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17338043833856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2509309419757056 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚548992102729728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151869996 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151879973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35285230 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28724678 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129995302 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64009634 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚16399Å ïfIÅ4Ñ2/ _wandbruntime‚13920  _runtime‚13920/é½}tÒ2‹N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_baïVÍ_&tch_end/inactive_split.all.current‚252 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2479303341752320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2496616280711695 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2509457673413632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28726426 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64013482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚566554516620288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2496615923122333 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17339056460800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130002934 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2479303324712960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚566554415729664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130002934 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1350 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281891973 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17339397807104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151889039 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17339397807104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28726423 trainer/global_step‚16399 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35287059 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281881980 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281891973 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130002932 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151879048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130002932 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2492118633992192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2509458031799296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2492118616952832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35287308 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64013482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚566554516620288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2496615923122333 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17339056460800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130002934 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2479303324712960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚566554415729664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130002934 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1350 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281891973 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17339397807104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151889039 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17339397807104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28726423 trainer/global_step‚16399 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35287059 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281881980 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281891973 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130002932 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151879048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879048 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130002932 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2492118633992192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2509458031799296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2492118616952832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35287308 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2479303341752320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2496616280711695 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2509457673413632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28726426 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976Æ ²Wn‘4Ô2/ _wandbruntime‚13920  _runtime‚13920³<ªÕ2ˆ /loss/FRt1/detection_loss‚0.4862924424009488 /loss/FRt2/detection_loss‚1.2772197687920988 1loss/SRqqt2/detection_loss‚0.6181758631827999 0loss/FBt1/detection_loss‚0.32862122539789573 %loss/total_loss‚7.165377908284224 epoch‚40 1loss/SRqqt1/assignment_loss‚0.312388412995834 2loss/FBt1/assignment_loss‚0.001762800654807465 2loss/FBt2/assignment_loss‚0.002422051223992352 1loss/SRqqt1/detection_loss‚0.8239271698400757 % _timestamp‚1.7842867285118532e+09 _runtime‚13920.637194854 _step‚1351 trainer/global_step‚16399 0loss/FRt1/assignment_loss‚1.4666252610443522 /loss/FBt2/detection_loss‚0.5862712677978714 0loss/FRt2/assignment_loss‚0.8156231550675518 3loss/SRqqt2/assignment_loss‚0.44604848988599616åÕ Ö2 trainer/global_step‚16399 0loss/FRt1/assignment_loss‚1.4666252610443522 /loss/FBt2/detection_loss‚0.5862712677978714 0loss/FRt2/assignment_loss‚0.8156231550675518 3loss/SRqqt2/assignment_loss‚0.44604848988599616 /loss/FRt1/detection_loss‚0.4862924424009488 /loss/FRt2/detection_loss‚1.2772197687920988 1loss/SRqqt2/detection_loss‚0.6181758631827999 0loss/FBt1/detection_loss‚0.32862122539789573 %loss/total_loss‚7.165377908284224 epoch‚40 1loss/SRqqt1/assignment_loss‚0.312388412995834 2loss/FBt1/assignment_loss‚0.001762800654807465 2loss/FBt2/assignment_loss‚0.002422051223992352 1loss/SRqqt1/detection_loss‚0.8239271698400757 % _timestamp‚1.7842867285118532e+09 _runtime‚13920.637194854 _step‚1351Ç DŽŒÆž×2‚(:“ –œèÒ«É€¤ gpu.0.gpu‚70.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝË8"&Ø2‚(:› –œèÒ›ÿÃ¥# disk.dm-0.out‚55174.44873046875$ memory_percent‚97.54705429077148 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.00390625! disk.rbd1.out‚118.76220703125 disk.sda1.out‚0 network.sent‚ 2.6055426e+07 disk.rbd3.in‚0$ disk./.usageGB‚545.6872978210449 disk.rbd3.out‚0& disk.nvme0n1.out‚55174.44873046875 disk.sda1.in‚0& proc.memory.availableMB‚ 401.890625) disk./.usagePercent‚36.62153241890992( disk.nvme0c0n1.out‚55174.44873046875 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 582.54296875# disk.nvme0n1.in‚144952.45703125% disk.nvme0c0n1.in‚144952.45703125 disk.dm-1.out‚ 582.54296875 proc.cpu.threads‚17 disk.sda2.in‚0 network.recv‚ 3.687828e+06 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚0.0625 disk.dm-0.in‚144952.45703125" proc.memory.rssMB‚ 2747.7734375 cpu‚26.014831849476455) proc.memory.percent‚16.77107810974121 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 582.54296875H£»Û4Ù2/ _wandbruntime‚13937  _runtime‚13937¦«‹Á%Ú2Š{ :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8476209044456482 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20009075105190277 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6854875087738037 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1041666716337204 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7842867456543074e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22727662324905396 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 epoch‚40 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.697783887386322 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20544078946113586 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2647944986820221 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2647944986820221 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952129125595093 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7328500151634216 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6361777186393738 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920614957809448 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6319100260734558 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5183333158493042 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.848284125328064 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5831087827682495 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5810173749923706 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2131495177745819 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04201188310980797 particle/accuracy_1_of_4‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2196855992078781 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8019874095916748 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976046681404114 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7660550475120544 *jet/accuracy_1_of_3‚0.9287372827529907 +jet/accuracy_3_of_4‚0.07847467809915543 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6812169551849365 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6371417045593262 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2750987410545349 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6038473844528198 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9928727149963379 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46329188346862793 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.639637291431427 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04527983441948891 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6583556532859802 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6647379994392395 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24009227752685547 particle/accuracy_2_of_2‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6601174473762512 particle/accuracy_1_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 *jet/accuracy_2_of_4‚0.8072261810302734 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27631860971450806 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6447821259498596 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5186167359352112 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5734824538230896 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9008379578590393 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6849584579467773 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01856350153684616 *jet/accuracy_1_of_4‚0.9804345369338989 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7713760137557983 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8766261339187622 6validation_average_jet_accuracy‚0.3755335211753845 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9932169914245605 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6023213267326355 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27489542961120605 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09173961728811264 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2442595660686493 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27208203077316284 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 _runtime‚13937.780567465 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.607954204082489 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8424168825149536 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2995944321155548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_3_of_3‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 particle/accuracy_5_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6095080375671387 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 /particle/accuracy_4_of_4‚0.9575355052947998 )loss/val_total_loss‚8.043241560981045 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2854556739330292 particle/accuracy_3_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.250864177942276 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9927153587341309 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6801020503044128 )jet/accuracy_2_of_2‚0.161927729845047 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_5‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_1_of_1‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019663507118821144 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.665730893611908 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.019749393686652184 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979740977287292 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6095080375671387 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24131372570991516 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6285168528556824 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043O¥ŒËÏ =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2489241659641266 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6509119868278503 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2697279453277588 +jet/accuracy_1_of_1‚0.33189669251441956 +particle/specificity‚0.9043875932693481 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23972772061824799 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9079392552375793 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_2_of_5‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.727357804775238 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6045778393745422 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4980016350746155 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6045960187911987 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6801020503044128 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.019489964470267296 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7957669496536255 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019617069512605667 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7785497307777405 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2521243691444397 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7237755656242371 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916567325592 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22609131038188934 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6591655015945435 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04831109195947647 ,jet/accuracy_4_of_4‚0.011098379269242287 'particle/f_score‚0.6266276240348816 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5425419807434082 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01856350153684616 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.764130711555481 +particle/sensitivity‚0.6158161759376526 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3614732623100281 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5842487215995789 particle/accuracy_2_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 trainer/global_step‚16399 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25919079780578613 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6321437954902649 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3821694552898407 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6279403567314148 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07944974303245544 +jet/accuracy_3_of_3‚0.05792704224586487 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961684346199036 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 _step‚1352 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5112120509147644 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6949906349182129 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6656317710876465 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8766261339187622 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20875194668769836 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22629694640636444 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36098524928092957 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.792508065700531 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33363908529281616 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9964523315429688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26554107666015625 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6333361268043518 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6624186635017395 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0371011383831501 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2568909227848053 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6223356127738953 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3223636746406555 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5585277080535889 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0\{ •=Û2{ 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2131495177745819 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04201188310980797 particle/accuracy_1_of_4‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2196855992078781 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8019874095916748 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.04629629850387573 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976046681404114 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7660550475120544 *jet/accuracy_1_of_3‚0.9287372827529907 +jet/accuracy_3_of_4‚0.07847467809915543 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6812169551849365 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6371417045593262 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2750987410545349 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6038473844528198 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9928727149963379 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46329188346862793 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.639637291431427 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04527983441948891 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.04629629850387573 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6583556532859802 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6647379994392395 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24009227752685547 particle/accuracy_2_of_2‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6601174473762512 particle/accuracy_1_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 *jet/accuracy_2_of_4‚0.8072261810302734 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27631860971450806 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6447821259498596 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5186167359352112 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5734824538230896 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9008379578590393 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6849584579467773 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01856350153684616 *jet/accuracy_1_of_4‚0.9804345369338989 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7713760137557983 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8766261339187622 6validation_average_jet_accuracy‚0.3755335211753845 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9932169914245605 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6023213267326355 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27489542961120605 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09173961728811264 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2442595660686493 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27208203077316284 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 _runtime‚13937.780567465 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.607954204082489 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8424168825149536 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2995944321155548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_3_of_3‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 particle/accuracy_5_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6095080375671387 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 /particle/accuracy_4_of_4‚0.9575355052947998 )loss/val_total_loss‚8.043241560981045 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2854556739330292 particle/accuracy_3_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.250864177942276 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9927153587341309 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6801020503044128 )jet/accuracy_2_of_2‚0.161927729845047 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_5‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_1_of_1‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019663507118821144 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.665730893611908 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.019749393686652184 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979740977287292 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6095080375671387 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24131372570991516 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6285168528556824 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976851761341095 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2489241659641266 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6509119868278503 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2697279453277588 +jet/accuracy_1_of_1‚0.33189669251441956 +particle/specificity‚0.9043875932693481 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23972772061824799 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9079392552375793 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 jet/accuracy_2_of_5‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.727357804775238 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6045778393745422 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4980016350746155 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6045960187911987 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6801020503044128 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.019489964470267296 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7957669496536255 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019617069512605667 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7785497307777405 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2521243691444397 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7237755656242371 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916567325592 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22609131038188934 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6591655015945435 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04831109195947647 ,jet/accuracy_4_of_4‚0.011098379269242287 'particle/f_score‚0.6266276240348816 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5425419807434082 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01856350153684616 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.764130711555481 +particle/sensitivity‚0.6158161759376526 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3614732623100281 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5842487215995789 particle/accuracy_2_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 trainer/global_step‚16399 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25919079780578613 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6321437954902649 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3821694552898407 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6279403567314148 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07944974303245544 +jet/accuracy_3_of_3‚0.05792704224586487 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961684346199036 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 _step‚1352 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5112120509147644 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6949906349182129 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6656317710876465 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8766261339187622 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20875194668769836 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22629694640636444 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36098524928092957 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.792508065700531 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33363908529281616 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9964523315429688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26554107666015625 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6333361268043518 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6624186635017395 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0371011383831501 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2568909227848053 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6223356127738953 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3223636746406555 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5585277080535889 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8476209044456482 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20009075105190277 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6854875087738037 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1041666716337204 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7842867456543074e+09 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22727662324905396 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 epoch‚40 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.697783887386322 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20544078946113586 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2647944986820221 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2647944986820221 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952129125595093 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7328500151634216 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6361777186393738 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920614957809448 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6319100260734558 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5183333158493042 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.848284125328064 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5831087827682495 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5810173749923706 particle/accuracy_1_of_2‚1È ¹;}ížÜ2‚(:“ ¥œèÒðƒ£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.354400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û&­|Ý2‚(:Ž ¥œèÒÜ⥠disk.sda1.out‚0 disk.sda.out‚ 583.3828125 proc.cpu.threads‚17 network.sent‚ 2.6118575e+07 disk.drbd2165.in‚0 disk.dm-1.out‚ 583.3828125 disk.sda.in‚ 0.0078125 disk.sda2.in‚0& proc.memory.availableMB‚ 398.734375# disk.nvme0n1.in‚144952.45703125 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625$ memory_percent‚97.56631851196289* disk./.usagePercent‚36.619961044664755 disk.rbd0.out‚0% disk.nvme0c0n1.out‚55177.91796875! disk.rbd1.out‚118.82470703125" proc.memory.rssMB‚ 2751.0703125* proc.memory.percent‚16.791200637817383 cpu‚26.015424957507484% disk.nvme0c0n1.in‚144952.45703125# disk.nvme0n1.out‚55177.91796875 disk.sda1.in‚0 disk.dm-0.out‚55177.91796875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 583.3828125 disk.dm-1.in‚ 0.0078125 disk.rbd3.out‚0 network.recv‚ 3.692477e+06$ disk./.usageGB‚545.6638832092285 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚144952.45703125œN“Þ2‚(:ˆ ´œèÒðûè¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰{€ß2‚(:” ´œèÒ³Œ¼¥ disk.sda1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚55188.22998046875 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 583.7265625$ memory_percent‚97.56584167480469* disk./.usagePercent‚36.619974357056535& disk.nvme0n1.out‚55188.22998046875 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 583.7265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚118.82470703125 disk.rbd3.out‚0 network.recv‚ 3.696354e+06 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚55188.22998046875% disk.nvme0c0n1.in‚144953.28515625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚144953.28515625 cpu‚26.01510367325211 network.sent‚ 2.6175847e+07 disk.sda.out‚ 583.7265625 disk.sda3.in‚ 0.0078125$ proc.memory.availableMB‚398.8125* proc.memory.percent‚16.791200637817383$ disk./.usageGB‚545.6640815734863# disk.nvme0n1.in‚144953.28515625 disk.rbd0.out‚0" proc.memory.rssMB‚ 2751.0703125 proc.cpu.threads‚17‡ð4à2/ _wandbruntime‚13977  _runtime‚13977¹¡ñ­Ìá2Æ _runtime‚13977.118352346 _step‚1353 ' lr-AdamW/pg1‚3.7258820355790895e-05 ' lr-AdamW/pg2‚3.7258820355790895e-05 trainer/global_step‚16449 % _timestamp‚1.7842867849927514e+09Â(cöÑâ2Ë ' lr-AdamW/pg1‚3.7258820355790895e-05 ' lr-AdamW/pg2‚3.7258820355790895e-05 trainer/global_step‚16449 % _timestamp‚1.7842867849927514e+09 _runtime‚13977.118352346 _step‚1353É û•$u4ã2/ _wandbruntime‚13977  _runtime‚13977ÿb.+áä2P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1354 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152404747 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2501262994055168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64232555 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130497566 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17597248243200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚13977.119626191 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2501262994055168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64232282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2518669466596352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17406472541184 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28814448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550679434143744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2501263011094528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17406813880320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568276682386944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17379923567788 % _timestamp‚1.7842867849938169e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WGDeviceStatsMonitor.on_train_batch_[*°& start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282902315 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35417834 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282892336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17406472541184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17380264117678 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17597330528768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152394770 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130497568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17406813880320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2518669466596352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2518669824974848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152404747 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169r¢ó¾ (å2†P JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17597330528768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152394770 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130497568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17406813880320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2518669466596352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2518669824974848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152404747 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1354 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152404747 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2501262994055168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64232555 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130497566 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17597248243200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚13977.119626191 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2501262994055168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64232282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2518669466596352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17406472541184 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28814448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550679434143744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2501263011094528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17406813880320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568276682386944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17379923567788 % _timestamp‚1.7842867849938169e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282902315 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35417834 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282892336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17406472541184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17380264117678 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438Ê ¦¤!4æ2/ _wandbruntime‚13977  _runtime‚13977"ve¶'ç2ŠN A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568311238158336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17598356643840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2518817720252928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17598272268288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64236130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282919013 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28816193 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550712862902272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2518818078638592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17407485168128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152403822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35419937 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550712881514496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2501410235084800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784286785605741e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282909020 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130505198 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282909020 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2505918456142150 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17381275171512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152413813 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130505198 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35420195 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152413813 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2518818078638592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17407826514432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2501410235084800 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2501410252124160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64236391 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282919013 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚16449 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17407826514432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚568311135170560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1355 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚13977.731544299,/7¤g%è2N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282909020 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2505918456142150 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17381275171512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152413813 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130505198 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35420195 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152413813 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2518818078638592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17407826514432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2501410235084800 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2501410252124160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64236391 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282919013 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚16449 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17407826514432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚568311135170560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1355 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚13977.731544299 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568311238158336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17598356643840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2518817720252928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17598272268288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64236130 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282919013 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28816193 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550712862902272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2518818078638592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17407485168128 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152403822 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35419937 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550712881514496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2501410235084800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784286785605741e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282909020 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130505198 E ì2‚(:’ ÃœèÒÙÆ®¢ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚145.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.177999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ,ಠí2‚(:ÿ ÃœèÒ·´×¥# disk.dm-0.out‚55194.80615234375* proc.memory.percent‚16.791200637817383 network.recv‚ 3.699957e+06 disk.sda3.out‚ 584.04296875! disk.rbd1.out‚118.88720703125 disk.rbd3.out‚0 disk.dm-0.in‚144954 disk.sda1.in‚0 network.sent‚ 2.6181035e+07 disk.nvme0c0n1.in‚144954 disk.rbd1.in‚0.0625 disk.rbd3.in‚0( proc.memory.availableMB‚ 398.25390625# disk./.usageGB‚545.663875579834 disk.sda.out‚ 584.04296875 disk.dm-1.in‚ 0.0078125$ memory_percent‚97.56925106048584 cpu‚26.01481932228431 proc.cpu.threads‚17 disk.rbd0.in‚0( disk.nvme0c0n1.out‚55194.80615234375 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.sda3.in‚ 0.0078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚55194.80615234375" proc.memory.rssMB‚ 2751.0703125 disk.nvme0n1.in‚144954 disk.sda1.out‚0* disk./.usagePercent‚36.619960532649685 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 584.04296875`S’{î2‚(:’ ÒœèÒžÉÍ¢ gpu.0.gpu‚28.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.925' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.785000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ħäï2‚(:‰ ÒœèÒê‡ñ¤ disk.sda1.out‚0 disk.drbd2165.out‚0 network.sent‚ 2.6244106e+07! disk.nvme0n1.in‚ 144954.828125 disk.sda1.in‚0" proc.memory.rssMB‚ 2751.0703125 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0( proc.memory.availableMB‚ 398.28515625 disk.rbd0.out‚0 disk.sda3.in‚ 0.0078125 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0.0625 cpu‚26.014503907199302 network.recv‚ 3.704283e+06$ disk./.usageGB‚545.6639518737793& disk.nvme0c0n1.out‚55203.861328125$ memory_percent‚97.56906032562256 disk.rbd0.in‚0! disk.rbd1.out‚118.88720703125# disk.nvme0c0n1.in‚ 144954.828125 disk.rbd3.out‚0! disk.dm-0.out‚55203.861328125$ disk.nvme0n1.out‚55203.861328125* proc.memory.percent‚16.791200637817383) disk./.usagePercent‚36.61996565280037 disk.sda.out‚ 584.234375 disk.sda3.out‚ 584.234375 disk.dm-1.out‚ 584.234375 disk.dm-0.in‚ 144954.828125eÙ3‘ð2‚(:† áœèÒƒ°¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ÀíGñ2‚(:’ áœèÒû€Û¤# disk.nvme0c0n1.in‚ 144954.828125 disk.sda3.out‚ 584.4765625 cpu‚26.014204508171314 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.rbd1.in‚0.0625* proc.memory.percent‚16.791200637817383 disk.sda2.in‚0 disk.dm-0.in‚ 144954.828125$ memory_percent‚97.56863117218018( proc.memory.availableMB‚ 398.35546875& disk.nvme0n1.out‚55207.34814453125 disk.drbd2165.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 network.sent‚ 2.6249132e+07) disk./.usagePercent‚36.61997819716955( disk.nvme0c0n1.out‚55207.34814453125 disk.sda.out‚ 584.4765625 disk.dm-1.out‚ 584.4765625" proc.memory.rssMB‚ 2751.0703125 disk.sda1.in‚0 disk.sda3.in‚ 0.0078125! disk.rbd1.out‚118.94970703125# disk.dm-0.out‚55207.34814453125 network.recv‚ 3.707869e+06! disk.nvme0n1.in‚ 144954.828125 disk.rbd0.out‚0 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0$ disk./.usageGB‚545.6641387939453 disk.drbd2165.out‚0&‘¶4ò2/ _wandbruntime‚14017  _runtime‚14017@„ªÊó2Ä trainer/global_step‚16499 % _timestamp‚1.7842868249798539e+09 _runtime‚14017.105442433 _step‚1357 & lr-AdamW/pg1‚3.624445591487589e-05 & lr-AdamW/pg2‚3.624445591487589e-05S&§AÏô2É & lr-AdamW/pg1‚3.624445591487589e-05 & lr-AdamW/pg2‚3.624445591487589e-05 trainer/global_step‚16499 % _timestamp‚1.7842868249798539e+09 _runtime‚14017.105442433 _step‚1357Í ׺­4õ2/ _wandbruntime‚14017  _runtime‚14017·eÅ(ö2€P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2495716544442368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17430540054696 trainer/global_step‚16499 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2526082213579776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2508625045536768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283738367 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842868249809053e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17457168043008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2526082213579776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2508625062576128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130879166 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚552350872070144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152849222 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35522264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28901701 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚569999628967936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152849222 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2526082571958272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚14017.106678887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17648738285568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2495716561481728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17457509382144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2526082571958272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17430880604586 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚569999530167296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130879168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152859199 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2508625045536768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2508625062576128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64423704 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2513147084497064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1358 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130879166 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17457509382144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152859199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚552350890682368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28901698 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17457168043008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17648658097152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976ÐÖ®· (÷2…P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64423704 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2513147084497064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1358 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130879166 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17457509382144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152859199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚552350890682368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28901698 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17457168043008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17648658097152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2495716544442368 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17430540054696 trainer/global_step‚16499 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2526082213579776 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2508625045536768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283738367 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842868249809053e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17457168043008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2526082213579776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2508625062576128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130879166 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚552350872070144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152849222 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35522264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28901701 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚569999628967936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152849222 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2526082571958272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚14017.106678887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17648738285568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2495716561481728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17457509382144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2526082571958272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17430880604586 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚569999530167296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130879168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152859199 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2508625045536768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2508625062576128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136Î œeß4ø2/ _wandbruntime‚14017  _runtime‚140171ب1ù2‹N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17458522016256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17649675343360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28903443 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283745072 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130886798 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚552384300828672 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283755065 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28903446 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152858274 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64427468 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842868255947137e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2508772286566400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2513294963812020 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17431551108418 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1359 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2526230825622016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2508772303605760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17458180669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2495863055114240 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17458522016256 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0„ÖXŽ'ú2N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283755065 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28903446 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152858274 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64427468 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842868255947137e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2508772286566400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2513294963812020 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17431551108418 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1359 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2526230825622016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2508772303605760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17458180669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2495863055114240 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17458522016256 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17458522016256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17649675343360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28903443 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283745072 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130886798 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚552384300828672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2503042078035968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153303674 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153313651 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131260768 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571722442407936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28988951 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35625777 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2520523234577572 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131260766 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17508204883968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2515987097018368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚554022328608768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35625509 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28988948 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2520523592166822 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131260768 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2533494960563200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284574419 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2503042078035968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153303674 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153313651 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131260768 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571722442407936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28988951 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35625777 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2520523234577572 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131260766 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17508204883968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2515987097018368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚554022328608768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35625509 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28988948 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2520523592166822 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131260768 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2533494960563200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284574419 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16549 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284581124 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571756773741056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚571756889311744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17509217518080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2520670756303166 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284581124 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2533643572605440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35627561 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2503188605747200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2516134338048000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚554055738755072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2516134355087360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284591117 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153322717 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2520671113892528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2533643572605440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17701131944448 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153322717 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64618518 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17509217518080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131268400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842868652996502e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17482508145328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17482167595326 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131268400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28990693 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚554055757367296 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2516134338048000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17508876171776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64618254 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438¶÷k¨'Œ3N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2516134338048000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚554055738755072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2516134355087360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284591117 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153322717 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2520671113892528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2533643572605440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17701131944448 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153322717 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64618518 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17509217518080 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131268400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842868652996502e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17482508145328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17482167595326 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131268400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28990693 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚554055757367296 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2516134338048000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17508876171776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64618254 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16549 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284581124 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571756773741056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚571756889311744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17509217518080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2520670756303166 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284581124 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2533643572605440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35627561 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2503188605747200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153768103 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573444995918336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2510367628668928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2523349165539328 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2540907707546624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17558900385792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35729242 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2527899384658080 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17558559046656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29076201 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚14096.765375712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555693766535168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2540908065925120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17751229383168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153758126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29076198 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131642366 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17558900385792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153758126 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555693747922944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17532113578402 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚16599 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2527899742247330 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153768103 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131642366 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35729518 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101È!Æ (™3†P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29076198 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131642366 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17558900385792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153758126 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555693747922944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17532113578402 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚16599 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2527899742247330 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153768103 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131642366 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35729518 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2540907707546624 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153768103 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573444995918336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2510367628668928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2523349165539328 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2540907707546624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17558900385792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35729242 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2527899384658080 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17558559046656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29076201 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚14096.765375712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555693766535168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2540908065925120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17751229383168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153758126Ö 5@Ø4š3/ _wandbruntime‚14097  _runtime‚14097FûS}'›3‹N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131650000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2523496406568960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842869052471828e+09 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153767178 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35731533 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17752248473600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17533124632236 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64809479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2528046906383674 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17559913019904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64809200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2523496406568960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285417176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131649998 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131650000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 trainer/global_step‚16599 _runtime‚14097.372971363 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285427169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚555727195293696 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285427169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2523496389529600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1367 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 285417176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29077946 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17559913019904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2510514139340800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2523496389529600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17559571673600 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚573479443767296 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2541056319588864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842869052471828e+09 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153767178 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35731533 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17752248473600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17533124632236 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64809479 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2528046906383674 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17559913019904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64809200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2523496406568960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285417176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131649998 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131650000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 trainer/global_step‚16599 _runtime‚14097.372971363 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285427169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚555727195293696 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285427169 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2523496389529600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1367 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 285417176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29077946 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17559913019904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2510514139340800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2523496389529600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17559571673600 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚573479443767296× "åkÃ43/ _wandbruntime‚14097  _runtime‚14097õPóž3‹ /loss/FRt2/detection_loss‚1.1881619673380512 0loss/FBt2/detection_loss‚0.47641320897734357 epoch‚41 2loss/FBt1/assignment_loss‚0.032862364538989566 1loss/SRqqt2/detection_loss‚0.5757600212083764 %loss/total_loss‚6.987699861749454 trainer/global_step‚16599 $ _timestamp‚1.784286905250486e+09 2loss/SRqqt1/assignment_loss‚0.3246726946212805 4loss/FBt2/assignment_loss‚0.00014872853569070088 /loss/FRt1/detection_loss‚0.5106273263975832 1loss/SRqqt1/detection_loss‚0.9308213211651102 0loss/FBt1/detection_loss‚0.35157935328321843 _runtime‚14097.375823748 _step‚1368 0loss/FRt1/assignment_loss‚1.4474253091111517 0loss/FRt2/assignment_loss‚0.7262936445689158 3loss/SRqqt2/assignment_loss‚0.42293392200374247™¨Ÿ3 2loss/SRqqt1/assignment_loss‚0.3246726946212805 4loss/FBt2/assignment_loss‚0.00014872853569070088 /loss/FRt1/detection_loss‚0.5106273263975832 1loss/SRqqt1/detection_loss‚0.9308213211651102 0loss/FBt1/detection_loss‚0.35157935328321843 _runtime‚14097.375823748 _step‚1368 0loss/FRt1/assignment_loss‚1.4474253091111517 0loss/FRt2/assignment_loss‚0.7262936445689158 3loss/SRqqt2/assignment_loss‚0.42293392200374247 /loss/FRt2/detection_loss‚1.1881619673380512 0loss/FBt2/detection_loss‚0.47641320897734357 epoch‚41 2loss/FBt1/assignment_loss‚0.032862364538989566 1loss/SRqqt2/detection_loss‚0.5757600212083764 %loss/total_loss‚6.987699861749454 trainer/global_step‚16599 $ _timestamp‚1.784286905250486e+09Ø .€‘ 3‚(:† »èÒçÁú¤ gpu.0.gpu‚48.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤¥m&¡3‚(:› »èÒ«§¨¥" proc.memory.rssMB‚ 2751.0703125 cpu‚26.012291474879422 proc.cpu.threads‚17 disk.sda3.out‚ 589.33203125 disk.rbd1.in‚0.0625! disk.rbd1.out‚119.19970703125 disk.dm-0.in‚144958.62890625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚55266.16357421875 disk.sda.in‚ 0.0078125 network.sent‚ 2.6395903e+07 disk.rbd3.out‚0# disk.dm-0.out‚55266.16357421875 disk.sda1.in‚0 disk.rbd0.in‚0$ memory_percent‚97.57106304168701$ disk./.usageGB‚545.6601753234863 disk.rbd3.in‚0% disk.nvme0c0n1.in‚144958.62890625 disk.sda3.in‚ 0.0078125 disk.sda1.out‚0 disk.sda.out‚ 589.33203125 disk.sda2.out‚0& disk.nvme0n1.out‚55266.16357421875 disk.sda2.in‚0* proc.memory.percent‚16.791200637817383 network.recv‚ 3.730672e+06 disk.dm-1.in‚ 0.0078125# disk.nvme0n1.in‚144958.62890625 disk.rbd0.out‚0( proc.memory.availableMB‚ 397.95703125) disk./.usagePercent‚36.61971220534147 disk.dm-1.out‚ 589.33203125 disk.drbd2165.in‚0Ï+螢3‚(:“ ÊèÒ¼Ÿû¢ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j¬Ê·"£3‚(:— ÊèÒ„ˆ”¥* proc.memory.percent‚16.791200637817383 proc.cpu.threads‚17* disk./.usagePercent‚36.619224510988744 disk.dm-1.in‚ 0.0078125 disk.dm-0.in‚144959.45703125" proc.memory.rssMB‚ 2751.0703125$ disk./.usageGB‚545.6529083251953' disk.nvme0c0n1.out‚55274.4130859375 disk.sda2.in‚0" disk.dm-0.out‚55274.4130859375$ memory_percent‚97.57256507873535 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0 cpu‚26.01199656198662# disk.nvme0n1.in‚144959.45703125 disk.sda1.in‚0 disk.dm-1.out‚ 589.53515625 disk.rbd3.out‚0' proc.memory.availableMB‚ 397.7109375% disk.nvme0n1.out‚55274.4130859375 disk.sda1.out‚0% disk.nvme0c0n1.in‚144959.45703125 disk.rbd1.in‚0.0625 network.recv‚ 3.735077e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚ 0.0078125! disk.rbd1.out‚119.19970703125 network.sent‚ 2.6459184e+07 disk.sda.out‚ 589.53515625 disk.sda3.out‚ 589.53515625«ˆRs ¤3‚(:• ÙèÒåÄŠ¤( disk.nvme0c0n1.out‚55283.61669921875 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 144960.171875 disk.sda.out‚ 589.76171875 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 589.76171875 disk.dm-0.in‚ 144960.171875" proc.memory.rssMB‚ 2751.0703125! disk.nvme0n1.in‚ 144960.171875 disk.rbd0.out‚0 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 397.96484375 disk.sda1.in‚0 disk.sda2.out‚0! disk.rbd1.out‚119.26220703125# disk.dm-0.out‚55283.61669921875 network.sent‚ 2.6464267e+07 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 disk.rbd3.out‚0$ memory_percent‚97.57101535797119 network.recv‚ 3.73865e+06* disk./.usagePercent‚36.619234239275045$ disk./.usageGB‚545.6530532836914 cpu‚26.011688234716594 proc.cpu.threads‚17& disk.nvme0n1.out‚55283.61669921875 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 589.76171875* proc.memory.percent‚16.791200637817383í[“¥3‚(:ˆ ÙèÒë¡ÿ£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚230.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.1152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çB—€4¦3/ _wandbruntime‚14136  _runtime‚14136Ï#¬Ì§3Æ ' lr-AdamW/pg1‚3.3278727370160694e-05 ' lr-AdamW/pg2‚3.3278727370160694e-05 trainer/global_step‚16649 % _timestamp‚1.7842869445882802e+09 _runtime‚14136.713863144 _step‚1369üøŇѨ3Ë _step‚1369 ' lr-AdamW/pg1‚3.3278727370160694e-05 ' lr-AdamW/pg2‚3.3278727370160694e-05 trainer/global_step‚16649 % _timestamp‚1.7842869445882802e+09 _runtime‚14136.713863144Ù þ|4©3/ _wandbruntime‚14136  _runtime‚14136 Þ(ª3€P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2548320812908544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286246523 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17609254548480 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2530711199981568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 $ _timestamp‚1.784286944589354e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154212578 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚557365204461568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17802315076608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17609595887616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2535275534738588 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2548320454530048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35832997 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575167618600960 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17582389515420 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64996718 _runtime‚14136.715111962 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1370 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2535275892327838 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575167500925952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2517693162262528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132023966 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚16649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154222555 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29163448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17802414139392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2548320454530048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286236544 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2548320812908544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64996445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29163451 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154212578 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17609595887616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132023968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088és’$à«3…P _runtime‚14136.715111962 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1370 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2535275892327838 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575167500925952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2517693162262528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132023966 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚16649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154222555 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29163448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17802414139392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2548320454530048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286236544 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2548320812908544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64996445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29163451 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_poolpö¢0+.freed‚ 154212578 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17609595887616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132023968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2548320812908544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286246523 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17609254548480 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2530711199981568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 $ _timestamp‚1.784286944589354e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154212578 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚557365204461568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17802315076608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17609595887616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2535275534738588 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2548320454530048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35832997 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575167618600960 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17582389515420 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64996718Ú U“À4¬3/ _wandbruntime‚14137  _runtime‚14137Êþ?O'­3ŠN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286263221 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286253228 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29165196 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2535423414053544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2530858441011200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784286945199743e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575201955229184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2535423056464182 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2548469066572288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65000449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17610267175424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35835253 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132031600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16649 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚14137.325534216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚557398614607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286253228 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65000185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2530858458050560 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286263221 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2517839672934400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132031598 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2530858458050560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2548469066572288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17610608521728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132031598 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17583741119144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17803431288320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2548468708186624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚557398633220096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154231621 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169ùõ~C'®3N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2535423056464182 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2548469066572288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65000449 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17610267175424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35835253 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132031600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16649 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚14137.325534216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚557398614607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286253228 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65000185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2530858458050560 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286263221 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2517839672934400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132031598 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2530858458050560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2548469066572288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17610608521728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132031598 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17583741119144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17803431288320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2548468708186624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚557398633220096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154231621 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286263221 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286253228 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29165196 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2535423414053544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2530858441011200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784286945199743e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚575201955229184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Û ¡6E4¯3/ _wandbruntime‚14137  _runtime‚14137sªÙ¦ °3‡ /loss/FBt1/detection_loss‚0.2004771927202318 _step‚1372 3loss/SRqqt2/assignment_loss‚0.41235800140415674 2loss/FBt1/assignment_loss‚0.006498117175656404 /loss/FRt1/detection_loss‚0.5029479847832159 trainer/global_step‚16649 _runtime‚ 14137.3286039 /loss/FRt2/detection_loss‚1.2122642243505901 $loss/total_loss‚6.60742560896937 0loss/FBt2/detection_loss‚0.20452102885771156 epoch‚41 0loss/FRt2/assignment_loss‚0.7389088610085575 2loss/SRqqt1/assignment_loss‚0.4321866451632118 3loss/FBt2/assignment_loss‚0.0015771348230016044 1loss/SRqqt1/detection_loss‚0.8043008923057406 1loss/SRqqt2/detection_loss‚0.6025924131741128 % _timestamp‚1.7842869452032745e+09 0loss/FRt1/assignment_loss‚1.4887931132031833uï‡6±3Œ 0loss/FBt2/detection_loss‚0.20452102885771156 epoch‚41 0loss/FRt2/assignment_loss‚0.7389088610085575 2loss/SRqqt1/assignment_loss‚0.4321866451632118 3loss/FBt2/assignment_loss‚0.0015771348230016044 1loss/SRqqt1/detection_loss‚0.8043008923057406 1loss/SRqqt2/detection_loss‚0.6025924131741128 % _timestamp‚1.7842869452032745e+09 0loss/FRt1/assignment_loss‚1.4887931132031833 /loss/FBt1/detection_loss‚0.2004771927202318 _step‚1372 3loss/SRqqt2/assignment_loss‚0.41235800140415674 2loss/FBt1/assignment_loss‚0.006498117175656404 /loss/FRt1/detection_loss‚0.5029479847832159 trainer/global_step‚16649 _runtime‚ 14137.3286039 /loss/FRt2/detection_loss‚1.2122642243505901 $loss/total_loss‚6.60742560896937Ü F}Fÿœ²3‚(:‘ èèÒö÷¢ gpu.0.gpu‚11.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fQý³3‚(:ˆ èèÒν¨¦* proc.memory.percent‚16.791200637817383# disk.dm-0.out‚55291.63427734375 disk.rbd0.out‚0! disk.nvme0n1.in‚ 144960.171875 disk.sda1.out‚0 disk.dm-1.in‚ 0.0078125( disk.nvme0c0n1.out‚55291.63427734375 cpu‚26.011385407730003# disk.nvme0c0n1.in‚ 144960.171875 network.sent‚ 2.64825e+07 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 397.44921875 disk.drbd2165.in‚0 disk.sda.out‚ 590.09375! disk.rbd1.out‚119.26220703125 disk.dm-0.in‚ 144960.171875 network.recv‚ 3.742352e+06& disk.nvme0n1.out‚55291.63427734375 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.57416248321533) disk./.usagePercent‚36.61974369426819" disk./.usageGB‚545.66064453125 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 590.09375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 590.09375" proc.memory.rssMB‚ 2751.0703125 disk.sda2.in‚0 disk.sda2.out‚0Ûî‚#“´3‚(:ˆ ÷èÒü½ó¢ gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ­* µ3‚(:ÿ ÷èÒ—»§¥ disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 590.55078125 disk.rbd3.in‚0 cpu‚26.011095631818026 disk.dm-0.in‚144961# disk.dm-0.out‚55299.84228515625& disk.nvme0n1.out‚55299.84228515625 disk.sda.out‚ 590.55078125 disk.rbd1.in‚0.0625 disk.nvme0n1.in‚144961 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 397.76171875 proc.cpu.threads‚17$ disk./.usageGB‚545.6607055664062 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚55299.84228515625 network.recv‚ 3.74644e+06 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 590.55078125 disk.sda1.out‚0* proc.memory.percent‚16.791200637817383 network.sent‚ 2.6531709e+07) disk./.usagePercent‚36.61974779038873 disk.sda1.in‚0 disk.nvme0c0n1.in‚144961$ memory_percent‚97.57225513458252" proc.memory.rssMB‚ 2751.0703125! disk.rbd1.out‚119.32470703125 disk.drbd2165.out‚0M¾ˆ“¶3‚(:ˆ †žèÒ¤÷Ë¢ gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚196.603' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.3206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$·3‚(:” †žèÒ¥Ëð¤ network.sent‚ 2.6536889e+07 disk.rbd3.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.in‚144961.71484375( proc.memory.availableMB‚ 397.76171875 disk.dm-1.out‚ 590.640625! disk.rbd1.out‚119.32470703125 network.recv‚ 3.749814e+06 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚55310.04248046875 disk.sda2.out‚0 cpu‚26.01078670810756 disk.sda3.in‚ 0.0078125# disk.dm-0.out‚55310.04248046875 disk.sda.out‚ 590.640625$ memory_percent‚97.57225513458252 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.dm-0.in‚144961.71484375) disk./.usagePercent‚36.61976468688599$ disk./.usageGB‚545.6609573364258 disk.sda3.out‚ 590.640625# disk.nvme0n1.in‚144961.71484375* proc.memory.percent‚16.791200637817383 proc.cpu.threads‚17& disk.nvme0n1.out‚55310.04248046875 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125" proc.memory.rssMB‚ 2751.0703125 disk.sda2.in‚0lµù›4¸3/ _wandbruntime‚14176  _runtime‚14176ÍÍE³Ê¹3Ä % _timestamp‚1.7842869843332565e+09 _runtime‚14176.458833629 _step‚1373 & lr-AdamW/pg1‚3.231615109638814e-05 & lr-AdamW/pg2‚3.231615109638814e-05 trainer/global_step‚16699 ÊávϺ3É _step‚1373 & lr-AdamW/pg1‚3.231615109638814e-05 & lr-AdamW/pg2‚3.231615109638814e-05 trainer/global_step‚16699 % _timestamp‚1.7842869843332565e+09 _runtime‚14176.458833629Ý ˜§Ë4»3/ _wandbruntime‚14176  _runtime‚14176­ð¢e¬¼3P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132405568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17660291389440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17633346552218 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2542652042408346 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2538073251463168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1374 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154667030 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2555733559891968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2538073268502528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287082575 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2542651684819096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17659950050304 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132405568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17659950050304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17853478427648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2555733201513472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287072596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2555733201513472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576890009432064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17633006002328 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35936542 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚14176.460036961 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚16699UI‡c (½3†P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132405566 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35936263 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2555733559891968 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132405566 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7842869843342984e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154667030 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29250698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559036642387968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154677007 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17853385656320 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2525018695856128 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2538073268502528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35936542 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚14176.460036961 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚16699 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132405568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17660291389440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17633346552218 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2542652042408346 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2538073251463168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1374 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154667030 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2555733559891968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2538073268502528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287082575 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2542651684819096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17659950050304 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132405568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17659950050304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17853478427648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2555733201513472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287072596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2555733201513472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576890009432064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17633006002328 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2525165189488640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2538220492492800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154686073 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154676082 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576924567833088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17660962677248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35938244 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17661304023552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576924462748160 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17634017056050 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65190687 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287099273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17854410213888 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2555881455170048 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2555881455170048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2525165189488640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2538220492492800 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154686073 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154676082 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576924567833088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17660962677248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35938244 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17661304023552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576924462748160 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17634017056050 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65190687 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287099273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17854410213888 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚578612602613760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚14216.472355165 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2563145948496896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2545435319984128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132787166 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2532344229449728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2545435302944768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132787166 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17710645552128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2563145948496896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155131459 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2563146306875392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2550027834899604 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚16749 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155131459 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578612709803008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132787168 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155121482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17904540911616 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚578612602613760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚14216.472355165 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2563145948496896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2545435319984128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132787166 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2532344229449728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2545435302944768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132787166 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17710645552128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2563145948496896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155131459 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2563146306875392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2550027834899604 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚16749 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155131459 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578612709803008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132787168 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155121482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17904540911616 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287935325 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132794800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17711999525376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65381674 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578647161805312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2545582561013760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚560741490460672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2550175356625198 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36042230 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65381926 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132794798 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2563294560539136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚1379 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29339693 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17905568356864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784287024954322e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17711999525376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29339696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2563294560539136 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2563294202153472 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155130534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚578647058817536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2545582561013760 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚16749 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17711658179072 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132794798 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36041981 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287935325 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚16749 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17711658179072 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132794798 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36041981 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287935325 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287935325 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132794800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17711999525376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65381674 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578647161805312 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2545582561013760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚560741490460672 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2550175356625198 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36042230 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65381926 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132794798 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2563294560539136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚1379 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29339693 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17905568356864 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 $ _timestamp‚1.784287024954322e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17711999525376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29339696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2563294560539136 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2563294202153472 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155130534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚578647058817536ã Rhr4Ñ3/ _wandbruntime‚14217  _runtime‚14217rxB¶Ò3‰ 0loss/FRt2/assignment_loss‚0.7321884201089758 3loss/SRqqt2/assignment_loss‚0.43371229174931225 3loss/FBt2/assignment_loss‚2.929290808924242e-05 3loss/SRqqt1/assignment_loss‚0.35741734779788853 -loss/FRt2/detection_loss‚1.17008887418187 1loss/SRqqt2/detection_loss‚0.5629512803548389 epoch‚41 4loss/FBt1/assignment_loss‚3.9023524155486676e-05 /loss/FRt1/detection_loss‚0.5084139085272464 %loss/total_loss‚6.819423883556524 trainer/global_step‚16749 % _timestamp‚1.7842870249569097e+09 _runtime‚14217.08225248 _step‚1380 0loss/FRt1/assignment_loss‚1.4473389040304583 1loss/SRqqt1/detection_loss‚0.8416814381028505 /loss/FBt1/detection_loss‚0.3990535194601127 /loss/FBt2/detection_loss‚0.3665095828107261»Å]-Ó3Ž 4loss/FBt1/assignment_loss‚3.9023524155486676e-05 /loss/FRt1/detection_loss‚0.5084139085272464 %loss/total_loss‚6.819423883556524 trainer/global_step‚16749 % _timestamp‚1.7842870249569097e+09 _runtime‚14217.08225248 _step‚1380 0loss/FRt1/assignment_loss‚1.4473389040304583 1loss/SRqqt1/detection_loss‚0.8416814381028505 /loss/FBt1/detection_loss‚0.3990535194601127 /loss/FBt2/detection_loss‚0.3665095828107261 0loss/FRt2/assignment_loss‚0.7321884201089758 3loss/SRqqt2/assignment_loss‚0.43371229174931225 3loss/FBt2/assignment_loss‚2.929290808924242e-05 3loss/SRqqt1/assignment_loss‚0.35741734779788853 -loss/FRt2/detection_loss‚1.17008887418187 1loss/SRqqt2/detection_loss‚0.5629512803548389 epoch‚41ä \Ã$é‘Ô3‚(:† ³žèÒî‘ù£ gpu.0.gpu‚52.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0赊ºÕ3‚(:‘ ³žèÒ¿šê¤! disk.rbd1.out‚119.51220703125 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 592.02734375$ memory_percent‚97.57452011108398 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 2.6609945e+07$ disk./.usageGB‚545.6623458862305 disk.rbd3.out‚0% disk.nvme0c0n1.out‚55324.86328125 disk.dm-1.in‚ 0.0078125& proc.memory.availableMB‚ 397.390625* proc.memory.percent‚16.791200637817383 disk.dm-0.out‚55324.86328125 disk.sda1.out‚0% disk.nvme0c0n1.in‚144961.80078125" proc.memory.rssMB‚ 2751.0703125* disk./.usagePercent‚36.619857873628455 disk.rbd1.in‚0.0625 disk.dm-0.in‚144961.80078125# disk.nvme0n1.in‚144961.80078125 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0# disk.nvme0n1.out‚55324.86328125 disk.sda2.in‚0 cpu‚26.009859116884112 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 592.02734375 disk.dm-1.out‚ 592.02734375 network.recv‚ 3.760629e+06 disk.sda2.out‚0—ŽþœÖ3‚(:‘ žèÒÒúТ gpu.0.gpu‚3.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð(,×3‚(:ø žèÒ¿Ôð¤ disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 55330.28125 disk.drbd2165.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 55330.28125 disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚545.6618499755859 disk.dm-0.in‚144962.00390625# disk.nvme0n1.in‚144962.00390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.0078125 disk.sda3.out‚592.3125# memory_percent‚97.5740909576416 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144962.00390625 disk.sda.out‚592.3125 disk.sda2.in‚0 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚592.3125! disk.rbd1.out‚119.51220703125" proc.memory.rssMB‚ 2751.0703125* proc.memory.percent‚16.791200637817383 cpu‚26.00956846955504 network.sent‚ 2.6673054e+07 disk.nvme0n1.out‚ 55330.28125' proc.memory.availableMB‚ 397.4609375 network.recv‚ 3.764983e+06' disk./.usagePercent‚36.619824592649?ºoE‘Ø3‚(:† ÑžèÒ¡ÿÌ¢ gpu.0.gpu‚43.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰±ö±%Ù3‚(:š ÑžèÒÛ ³¤ disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.rbd3.out‚0 disk.dm-0.in‚144962.07421875# disk.dm-0.out‚55353.78271484375# disk.nvme0n1.in‚144962.07421875$ memory_percent‚97.57411479949951 disk.sda1.in‚0( disk.nvme0c0n1.out‚55353.78271484375" proc.memory.rssMB‚ 2751.0703125 cpu‚26.00926295342675& disk.nvme0n1.out‚55353.78271484375 disk.drbd2165.out‚0* proc.memory.percent‚16.791200637817383 disk.rbd3.in‚0$ disk./.usageGB‚536.2213821411133 disk.sda3.out‚ 592.69140625 network.recv‚ 3.768475e+06 disk.rbd1.in‚0.0625% disk.nvme0c0n1.in‚144962.07421875( proc.memory.availableMB‚ 397.45703125) disk./.usagePercent‚35.98626687519743 disk.dm-1.out‚ 592.69140625! disk.rbd1.out‚119.57470703125 disk.sda1.out‚0 disk.sda.out‚ 592.69140625 proc.cpu.threads‚17 network.sent‚ 2.6678232e+07 disk.rbd0.in‚0 disk.rbd0.out‚0w|À4Ú3/ _wandbruntime‚14256  _runtime‚14256b¸UÌÛ3Æ _runtime‚14256.237513437 _step‚1381 ' lr-AdamW/pg1‚3.0430370607892718e-05 ' lr-AdamW/pg2‚3.0430370607892718e-05 trainer/global_step‚16799 % _timestamp‚1.7842870641119447e+09ÛÄPöÑÜ3Ë ' lr-AdamW/pg1‚3.0430370607892718e-05 ' lr-AdamW/pg2‚3.0430370607892718e-05 trainer/global_step‚16799 % _timestamp‚1.7842870641119447e+09 _runtime‚14256.237513437 _step‚1381å M¬#Û4Ý3/ _wandbruntime‚14256  _runtime‚14256fu™,µÞ3P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580335106770944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚14256.238750011 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1382 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133168768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17761682393088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133168766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562379499628544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17955704108032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2570559053858816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2539669746003968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155575934 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155575934 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2557404342569362 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17761682393088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17761341053952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2552797371465728 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36143423 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7842870641129906e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133168768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29425198 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2557403984980112 YKDeviceStatsMonitor.on_train_batch_st•æã×R art/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2552797371465728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17761341053952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562379518240768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580335222348800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2552797354426368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288754679 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65568621 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2570558695480320 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155585911 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2570558695480320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133168766Ýó2ˆ (ß3†P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133168766 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580335106770944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _runtime‚14256.238750011 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1382 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133168768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17761682393088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133168766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562379499628544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17955704108032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2570559053858816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2539669746003968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155575934 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155575934 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2557404342569362 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17761682393088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17761341053952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2552797371465728 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36143423 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7842870641129906e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133168768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29425198 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2557403984980112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2552797371465728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17761341053952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562379518240768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚580335222348800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2552797354426368 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288754679 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65568621 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2570558695480320 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155585911 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2570558695480320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360æ ÀZV4à3/ _wandbruntime‚14256  _runtime‚14256Sî{¦'á3ŠN HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚16799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29426943 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2570707307522560 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2552944612495360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2539816273715200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17762695027200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17762695027200 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17762353680896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2557551506705706 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288761384 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133176400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562412928387072 _step‚1383 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2552944595456000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155594977 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155584986 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2539816256675840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288771377 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288761384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 288771377 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65572425 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2557551864295068 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2570706949136896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65572702dkî¤;&â3N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2570706949136896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65572702 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚16799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29426943 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2570707307522560 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2552944612495360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2539816273715200 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17762695027200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17762695027200 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17762353680896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2557551506705706 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288761384 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133176400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562412928387072 _step‚1383 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2552944595456000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155594977 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155584986 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2539816256675840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288771377 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288761384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 288771377 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65572425 ZMDeviceStatsMonitor.on_train_batch_end/inactive—&ÂÚ_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2557551864295068 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362ç 4hÓ4ã3/ _wandbruntime‚14256  _runtime‚14256­=´`ä3‰ 2loss/SRqqt2/assignment_loss‚0.4541002578585866 /loss/FRt2/detection_loss‚1.2888540092539482 1loss/SRqqt1/detection_loss‚0.9906181222635798 /loss/FBt1/detection_loss‚0.3425616309480794 epoch‚41 trainer/global_step‚16799 % _timestamp‚1.7842870647292254e+09 4loss/FBt2/assignment_loss‚3.0306273636973365e-06 /loss/FRt1/detection_loss‚0.4707092741906234 %loss/total_loss‚7.226110033304739 _runtime‚14256.854560283 _step‚1384 1loss/SRqqt2/detection_loss‚0.5745304526816483 /loss/FBt2/detection_loss‚0.3949520275035376 0loss/FRt2/assignment_loss‚0.9236116809734507 2loss/SRqqt1/assignment_loss‚0.2770841431697685 2loss/FBt1/assignment_loss‚0.012009505742892411 0loss/FRt1/assignment_loss‚1.4970758980912606Þ Û‡å3Ž 0loss/FRt1/assignment_loss‚1.4970758980912606 2loss/SRqqt2/assignment_loss‚0.4541002578585866 /loss/FRt2/detection_loss‚1.2888540092539482 1loss/SRqqt1/detection_loss‚0.9906181222635798 /loss/FBt1/detection_loss‚0.3425616309480794 epoch‚41 trainer/global_step‚16799 % _timestamp‚1.7842870647292254e+09 4loss/FBt2/assignment_loss‚3.0306273636973365e-06 /loss/FRt1/detection_loss‚0.4707092741906234 %loss/total_loss‚7.226110033304739 _runtime‚14256.854560283 _step‚1384 1loss/SRqqt2/detection_loss‚0.5745304526816483 /loss/FBt2/detection_loss‚0.3949520275035376 0loss/FRt2/assignment_loss‚0.9236116809734507 2loss/SRqqt1/assignment_loss‚0.2770841431697685 2loss/FBt1/assignment_loss‚0.012009505742892411è ÞÛ8ú“æ3‚(:ˆ àžèÒý·¬¤ gpu.0.gpu‚56.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h?®å ç3‚(:• àžèÒ¨¹º¦ disk.sda2.in‚0 disk.sda3.out‚ 592.90234375! disk.rbd1.out‚119.57470703125 cpu‚26.01113215776948 disk.dm-0.in‚144962.07421875" disk.dm-0.out‚55410.7978515625 network.sent‚ 2.668325e+07 network.recv‚ 3.771603e+06$ disk./.usageGB‚536.2195014953613 disk.rbd0.in‚0' disk.nvme0c0n1.out‚55410.7978515625 disk.rbd1.in‚0.0625 disk.rbd3.in‚0) disk./.usagePercent‚35.98614066348305 disk.sda1.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.855478286743164 disk.dm-1.out‚ 592.90234375% disk.nvme0c0n1.in‚144962.07421875 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125" proc.memory.rssMB‚ 2761.6015625# disk.nvme0n1.in‚144962.07421875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125 disk.sda.out‚ 592.90234375 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚97.63655662536621' proc.memory.availableMB‚ 387.2265625 proc.cpu.threads‚17% disk.nvme0n1.out‚55410.7978515625 disk.rbd3.out‚0NT´§4è3/ _wandbruntime‚14274  _runtime‚14274tÞ·ˆ=é3‚{ >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9180835485458374 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 (particle/accuracy‚0.8422619700431824 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6299911737442017 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9579713940620422 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6260575652122498 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37290769815444946 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24600167572498322 ,jet/accuracy_3_of_3‚0.057598523795604706 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5727987885475159 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5279786586761475 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6122625470161438 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01892995461821556 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880818128585815 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04354191571474075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9876542091369629 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7228721976280212 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6348696947097778 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.657974898815155 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950240850448608 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 trainer/global_step‚16799 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7360544204711914 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27618125081062317 *jet/accuracy_1_of_1‚0.3327600657939911 +particle/sensitivity‚0.6169867515563965 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8571711778640747 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7248677015304565 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2781335413455963 ( Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5 *jet/accuracy_1_of_4‚0.9791117310523987 *jet/accuracy_2_of_4‚0.7995047569274902 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25167393684387207 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.733711302280426 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.019136736169457436 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36210760474205017 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2256363034248352 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.694050669670105 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6638209223747253 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6185714602470398 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.019784104079008102 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7350906729698181 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6051337122917175 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6107739210128784 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25229912996292114 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21748985350131989 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8017765283584595 )loss/val_total_loss‚8.061611729736603 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2393045872449875 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.594002366065979 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.043144673109054565 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3003740906715393 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6335232853889465 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5471962690353394 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_3_of_4‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5317040681838989 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7762305736541748 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27597612142562866 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6657775044441223 % _timestamp‚1.7842870820242841e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6977843642234802 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416666626930237 )particle/specificity‚0.90386962890625 particle/accuracy_4_of_5‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6261518597602844 particle/accuracy_1_of_3‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04885254055261612 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2831275463104248 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6628680229187012 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.585725724697113 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5376830101013184 particle/accuracy_2_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26082128286361694 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚14274.150510976 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884513020515442 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6122625470161438 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6036237478256226 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26755425333976746 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148059844971 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24049381911754608 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7687435150146484 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942079186439514 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019269784912467003 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3621181547641754 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21076644957065582 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6092140078544617 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20909278094768524 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6719387769699097 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5784044861793518 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6046140789985657 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2658989429473877 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7955046892166138 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7927395701408386 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2580384612083435 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08240505307912827 +jet/accuracy_2_of_2‚0.15952086448669434 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7629290819168091 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33417680859565735 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09351132810115814 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.635043203830719 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7197638154029846 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2658989429473877 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21178500354290009 particle/accuracy_5_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2335113137960434 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3757856488227844 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01892995461821556 particle/accuracy_2_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4924601912498474 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_3_of_4‚0.080002561211586 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29616647958755493 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8979318737983704 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6334512233734131 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6636737585067749 ,jet/accuracy_4_of_4‚0.011098379269242287 'particle/f_score‚0.6268343329429626 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7360544204711914 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8700957298278809 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45253103971481323 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2734724283218384 *jet/accuracy_2_of_3‚0.5760686993598938 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6664979457855225 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6517459750175476 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7295044660568237 epoch‚41 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8469921946525574 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7630088329315186 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.984316885471344 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2005893886089325 *jet/accuracy_1_of_3‚0.9267323613166809 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24382495880126953 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035915013402700424 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_5‚1 _step‚1385 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32338425517082214 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6272770762443542 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8700957298278809 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019648339599370956 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458134651184 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2412991225719452 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20236285030841827øMx!5ê3‡{ particle/accuracy_1_of_3‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04885254055261612 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2831275463104248 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6628680229187012 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.585725724697113 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5376830101013184 particle/accuracy_2_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26082128286361694 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚14274.150510976 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884513020515442 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6122625470161438 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6036237478256226 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26755425333976746 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148059844971 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24049381911754608 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7687435150146484 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942079186439514 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.019269784912467003 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3621181547641754 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21076644957065582 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6092140078544617 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20909278094768524 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6719387769699097 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5784044861793518 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6046140789985657 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2658989429473877 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7955046892166138 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7927395701408386 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2580384612083435 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08240505307912827 +jet/accuracy_2_of_2‚0.15952086448669434 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7629290819168091 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33417680859565735 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09351132810115814 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.635043203830719 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7197638154029846 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2658989429473877 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21178500354290009 particle/accuracy_5_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2335113137960434 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3757856488227844 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01892995461821556 particle/accuracy_2_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4924601912498474 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_3_of_4‚0.080002561211586 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29616647958755493 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8979318737983704 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6334512233734131 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6636737585067749 ,jet/accuracy_4_of_4‚0.011098379269242287 'particle/f_score‚0.6268343329429626 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7360544204711914 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8700957298278809 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45253103971481323 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2734724283218384 *jet/accuracy_2_of_3‚0.5760686993598938 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6664979457855225 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6517459750175476 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7295044660568237 epoch‚41 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8469921946525574 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7630088329315186 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.984316885471344 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2005893886089325 *jet/accuracy_1_of_3‚0.9267323613166809 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24382495880126953 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035915013402700424 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_5‚1 _step‚1385 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32338425517082214 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6272770762443542 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8700957298278809 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.019648339599370956 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458134651184 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2412991225719452 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20236285030841827 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9180835485458374 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 (particle/accuracy‚0.8422619700431824 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6299911737442017 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9579713940620422 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6260575652122498 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37290769815444946 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24600167572498322 ,jet/accuracy_3_of_3‚0.057598523795604706 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5727987885475159 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5279786586761475 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6122625470161438 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01892995461821556 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880818128585815 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04354191571474075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9876542091369629 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7228721976280212 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6348696947097778 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.657974898815155 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950240850448608 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 trainer/global_step‚16799 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7360544204711914 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27618125081062317 *jet/accuracy_1_of_1‚0.3327600657939911 +particle/sensitivity‚0.6169867515563965 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8571711778640747 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7248677015304565 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2781335413455963 ( Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5 *jet/accuracy_1_of_4‚0.9791117310523987 *jet/accuracy_2_of_4‚0.7995047569274902 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25167393684387207 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.733711302280426 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.019136736169457436 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36210760474205017 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2256363034248352 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.694050669670105 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6638209223747253 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6185714602470398 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.019784104079008102 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7350906729698181 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6051337122917175 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6107739210128784 @&Purity/0FRt2SRqqt1FBt/event_pr Dvloportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25229912996292114 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21748985350131989 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8017765283584595 )loss/val_total_loss‚8.061611729736603 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2393045872449875 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.594002366065979 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.043144673109054565 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3003740906715393 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6335232853889465 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5471962690353394 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_3_of_4‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5317040681838989 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7762305736541748 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27597612142562866 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6657775044441223 % _timestamp‚1.7842870820242841e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6977843642234802 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416666626930237 )particle/specificity‚0.90386962890625 particle/accuracy_4_of_5‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6261518597602844é IïØÓë3‚(:’ ïžèÒ§²ï¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.392' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.27839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pöîÏì3‚(:” ïžèÒšñû¤$ disk./.usageGB‚536.2239875793457 disk.sda.in‚ 0.0078125! disk.rbd1.out‚119.63720703125 cpu‚26.013429726306946 disk.dm-1.in‚ 0.0078125 network.recv‚ 3.776168e+06# disk.nvme0n1.in‚144962.07421875 disk.sda1.out‚0( disk.nvme0c0n1.out‚55419.20068359375* proc.memory.percent‚16.919946670532227& disk.nvme0n1.out‚55419.20068359375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 593.1484375 disk.rbd1.in‚0.0625" proc.memory.rssMB‚ 2772.1640625# disk.dm-0.out‚55419.20068359375 disk.sda3.out‚ 593.1484375 network.sent‚ 2.6746366e+07 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144962.07421875 disk.sda.out‚ 593.1484375$ memory_percent‚97.70393371582031$ proc.memory.availableMB‚376.1875 disk.dm-0.in‚144962.07421875 disk.sda1.in‚0 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0) disk./.usagePercent‚35.98644172834332 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0ë- êží3‚(:“ þžèÒÀÇž¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.461799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à¬Vî3‚(:“ þžèÒվݤ disk.sda3.out‚ 593.3828125! disk.rbd1.out‚119.63720703125" proc.memory.rssMB‚ 2772.3828125 cpu‚26.013085362352104 disk.sda.out‚ 593.3828125 disk.sda2.in‚0 disk.rbd1.in‚0.0625) disk./.usagePercent‚35.98681601135838 disk.dm-0.in‚144962.07421875" disk.dm-0.out‚55427.6650390625% disk.nvme0n1.out‚55427.6650390625 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 593.3828125% disk.nvme0c0n1.in‚144962.07421875 disk.sda.in‚ 0.0078125 proc.cpu.threads‚17# disk./.usageGB‚536.229564666748# disk.nvme0n1.in‚144962.07421875 disk.sda3.in‚ 0.0078125 network.sent‚ 2.6803674e+07 disk.sda1.out‚0' disk.nvme0c0n1.out‚55427.6650390625 disk.sda2.out‚0 disk.rbd3.out‚0$ memory_percent‚97.70407676696777 disk.rbd3.in‚0 network.recv‚ 3.780306e+06' proc.memory.availableMB‚ 376.1640625 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.921281814575195 disk.drbd2165.out‚0£™Äžï3‚(:“ ŸèÒç¾Ë¢ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚262.342' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.468399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾¶ã1$ð3‚(:™ ŸèÒ•Èû¤$ disk./.usageGB‚536.2367210388184% disk.nvme0c0n1.in‚144962.07421875 disk.sda.out‚ 593.52734375 disk.dm-1.out‚ 593.52734375! disk.rbd1.out‚119.69970703125 disk.rbd3.in‚0# disk.nvme0n1.in‚144962.07421875 disk.sda1.out‚0 cpu‚26.012809568278705 disk.drbd2165.in‚0# disk.dm-0.out‚55439.31201171875" proc.memory.rssMB‚ 2772.3828125) disk./.usagePercent‚35.98729628149261 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125$ memory_percent‚97.70402908325195* proc.memory.percent‚16.921281814575195 network.recv‚ 3.783618e+06 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 593.52734375 proc.cpu.threads‚17 disk.dm-0.in‚144962.07421875 network.sent‚ 2.6808725e+07& disk.nvme0n1.out‚55439.31201171875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125( disk.nvme0c0n1.out‚55439.31201171875 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.rbd3.out‚0& proc.memory.availableMB‚ 376.171875åK¥Ö4ñ3/ _wandbruntime‚14313  _runtime‚14313Çñ,Êò3Ä _runtime‚14313.488951539 _step‚1386 & lr-AdamW/pg1‚2.950728751332686e-05 & lr-AdamW/pg2‚2.950728751332686e-05 trainer/global_step‚16849 % _timestamp‚1.7842871213633628e+09]©tÍÏó3É & lr-AdamW/pg1‚2.950728751332686e-05 & lr-AdamW/pg2‚2.950728751332686e-05 trainer/global_step‚16849 % _timestamp‚1.7842871213633628e+09 _runtime‚14313.488951539 _step‚1386ê ¸z‚‚4ô3/ _wandbruntime‚14313  _runtime‚14313·O±§(õ3P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133671034 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2562088989597696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17829769761280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289781719 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18025057920000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2579918742319616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29514971 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29514968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2566706875589179 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289771740 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156110685 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156100708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36277310 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2562088972558336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17830111100416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2579919100698112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842871213644044e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582091818997760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17802915738171 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156110685 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2548903942811648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17829769761280 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17830111100416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133671034 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2562088989597696 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17829769761280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289781719 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18025057920000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2579918742319616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29514971 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29514968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2566706875589179 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289771740 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156110685 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156100708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36277310 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2562088972558336 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17830111100416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2579919100698112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842871213644044e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582091818997760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17802915738171 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156110685 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2548903942811648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17829769761280 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17830111100416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65795733 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29516713 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17831123734528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156119751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65796019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582126374256640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29516716 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582126264977408 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17831123734528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289798417 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚564100278413312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18025986564096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133678666 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17803586242003 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ;4DeviceStatsMonitor.on„¶¢Øh_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2549050470522880 _runtime‚14314.102479536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36279020 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842871219765537e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289788424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133678666 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2562236230627328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2562236213587968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156119751 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2562236213587968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133678664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _step‚1388 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2580067354361856 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17830782388224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133678664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289798417 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2580066995976192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17830782388224 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2562236230627328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2691=hž'ù3N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289798417 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2580066995976192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17830782388224 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2562236230627328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289788424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65795733 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29516713 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17831123734528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156119751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65796019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582126374256640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29516716 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582126264977408 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17831123734528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289798417 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚564100278413312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18025986564096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133678666 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17803586242003 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2549050470522880 _runtime‚14314.102479536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36279020 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842871219765537e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289788424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133678666 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2562236230627328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2562236213587968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156119751 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2562236213587968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133678664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _step‚1388 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2580067354361856 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17830782388224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133678664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144ì 7ÉVC4ú3/ _wandbruntime‚14314  _runtime‚14314w‹åû3‰ 1loss/SRqqt1/detection_loss‚0.8016620401265587 _runtime‚14314.105133736 0loss/FRt1/assignment_loss‚1.3967636908061625 /loss/FRt2/assignment_loss‚0.827403675107395 3loss/SRqqt1/assignment_loss‚0.32912034162341086 3loss/FBt1/assignment_loss‚0.0024011770275962907 3loss/FBt2/assignment_loss‚0.0007544046275275004 1loss/SRqqt2/detection_loss‚0.5849109535142861 epoch‚42 2loss/SRqqt2/assignment_loss‚0.4177889727829675 /loss/FRt1/detection_loss‚0.4857570445372914 0loss/FBt2/detection_loss‚0.42899655192949077 trainer/global_step‚16849 /loss/FBt1/detection_loss‚0.4216164641549369 %loss/total_loss‚6.939541941039872 % _timestamp‚1.7842871219794734e+09 _step‚1389 .loss/FRt2/detection_loss‚1.242366624802248ÛSü3Ž 0loss/FRt1/assignment_loss‚1.3967636908061625 /loss/FRt2/assignment_loss‚0.827403675107395 3loss/SRqqt1/assignment_loss‚0.32912034162341086 3loss/FBt1/assignment_loss‚0.0024011770275962907 3loss/FBt2/assignment_loss‚0.0007544046275275004 1loss/SRqqt2/detection_loss‚0.5849109535142861 epoch‚42 2loss/SRqqt2/assignment_loss‚0.4177889727829675 /loss/FRt1/detection_loss‚0.4857570445372914 0loss/FBt2/detection_loss‚0.42899655192949077 trainer/global_step‚16849 /loss/FBt1/detection_loss‚0.4216164641549369 %loss/total_loss‚6.939541941039872 % _timestamp‚1.7842871219794734e+09 _step‚1389 .loss/FRt2/detection_loss‚1.242366624802248 1loss/SRqqt1/detection_loss‚0.8016620401265587 _runtime‚14314.105133736í à°k?žý3‚(:“ œŸèÒÿêò¢ gpu.0.gpu‚55.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Ìw6þ3‚(:” œŸèÒÉ똦$ disk./.usageGB‚536.2426643371582 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚144962.07421875 disk.dm-1.out‚ 593.80859375" proc.memory.rssMB‚ 2772.3828125 network.recv‚ 3.786927e+06 disk.rbd3.out‚0% disk.nvme0n1.out‚55453.7431640625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 593.80859375$ memory_percent‚97.70545959472656 disk.dm-0.in‚144962.07421875 disk.sda1.out‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125$ proc.memory.availableMB‚375.9375 disk.rbd3.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.921281814575195 disk.rbd1.in‚0.0625! disk.rbd1.out‚119.69970703125 proc.cpu.threads‚17) disk./.usagePercent‚35.98769514123096" disk.dm-0.out‚55453.7431640625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚55453.7431640625 cpu‚26.012495812070668 network.sent‚ 2.6826956e+07 disk.sda.in‚ 0.0078125 disk.sda.out‚ 593.80859375 disk.sda2.out‚0# disk.nvme0n1.in‚144962.07421875 disk.sda2.in‚0?DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156555160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565738287581184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134052634 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2569451041079296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134052634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2587331489303040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565738306193408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2569451041079296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2569451024039936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17880465263104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18076435010048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17853191675189 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2587331847681536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36380917 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17853532225079 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290617771 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156555160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚14353.413552883 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583814821391872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65982876 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134052632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65983138 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17880806602240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979lÎÕd@ †4ƒP C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1391 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583814722591232 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18076515198464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17880465263104 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2587331847681536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2574083025669687 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2587331489303040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17880806602240 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156555160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565738287581184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134052634 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2569451041079296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134052634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2587331489303040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565738306193408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2569451041079296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2569451024039936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17880465263104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18076435010048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17853191675189 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2587331847681536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36380917 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17853532225079 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290617771 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156555160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚14353.413552883 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583814821391872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65982876 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134052632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65983138 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17880806602240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591ï iG‹4‡4/ _wandbruntime‚14354  _runtime‚14354"«pA'ˆ4ŠN \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583849281524736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29603966 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17854202728911 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134060264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565771734951936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18077468488704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842871618998802e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2569598265069568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17881477890048 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65986729 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36382766 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290634469 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583849184828416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290634469 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134060266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16899 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29603963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36383010 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2587480101345280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290624476 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156564212 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134060264¥ty"'‰4N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156574203 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2556376004116480 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17854543278913 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17881819236352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156564212 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2574230189806031 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2587480101345280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17881477890048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18077546572800 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2556375987077120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _step‚1392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2574230547395393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2569598265069568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290624476 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚14354.025685854 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16899 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29603963 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36383010 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2587480101345280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290624476 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156564212 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134060264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583849281524736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29603966 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17854202728911 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134060264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565771734951936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18077468488704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842871618998802e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2569598265069568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17881477890048 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65986729 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36382766 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290634469 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583849184828416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290634469 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134060266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29689471 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚14393.443540098 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18127821638144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2563555009998848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157019589 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157019589 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2576813092560896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17931160764928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17931502104064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66174182 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17904148711987 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17931502104064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36484975 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29689468 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134434234 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2594744594664960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157009612 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36484714 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291453823 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842872013169677e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2594744594664960 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2576813075521536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2594744236286464 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2576813075521536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚16949 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2594744236286464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚585537469054464 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977ˆfE (˜4…P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2563555027038208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚16949 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2594744236286464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚585537469054464 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2581459175750195 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚1395 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134434232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17931160764928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29689471 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚14393.443540098 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18127821638144 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2563555009998848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157019589 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157019589 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2576813092560896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17931160764928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17931502104064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66174182 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17904148711987 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17931502104064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36484975 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29689468 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134434234 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2594744594664960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157009612 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36484714 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291453823 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 % _timestamp‚1.7842872013169677e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2594744594664960 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2576813075521536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2594744236286464 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2576813075521536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17932173391872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17905159765821 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2576960316551168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18128845270528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2563701537710080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291470521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚16949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17932173391872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291460528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134441866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585572018148864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2563701520670720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29691216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18128752506368 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1396 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291460528 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29691213 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚585571906772480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2594892848328704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66178254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚567443172878336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291470521 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134441864 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36487038 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2576960333590528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17932514738176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚14394.055653996 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157028655 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2576960333590528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17904819215819 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157018664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157018664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66177986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169$Ï·&›4N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚14394.055653996 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157028655 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 H@DeviceStatsMonitor.on_train_batch_end/segment.é"˃†%small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2576960333590528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17904819215819 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157018664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157018664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66177986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134441866 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2594892848328704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2581606697475901 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842872019298012e+09 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17932173391872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17905159765821 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2576960316551168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18128845270528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2563701537710080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291470521 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚16949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17932173391872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291460528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134441866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585572018148864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2563701520670720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29691216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18128752506368 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1396 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291460528 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29691213 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚585571906772480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2594892848328704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66178254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚567443172878336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291470521 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134441864 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36487038 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2576960333590528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17932514738176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ô Ù”µ4œ4/ _wandbruntime‚14394  _runtime‚14394"ã à4‰ 0loss/FRt2/assignment_loss‚0.7878845096506192 2loss/FBt2/assignment_loss‚0.002465879553582795 0loss/FBt1/detection_loss‚0.37839067656823316 %loss/total_loss‚7.158504920841336 3loss/SRqqt1/assignment_loss‚0.36316494998515764 2loss/SRqqt2/assignment_loss‚0.4358955210990837 3loss/FBt1/assignment_loss‚9.182144265450023e-05 /loss/FRt1/detection_loss‚0.5218292850963414 0loss/FRt1/assignment_loss‚1.4312736665681922 0loss/SRqqt1/detection_loss‚0.963595663408204 trainer/global_step‚16949 % _timestamp‚1.7842872019315906e+09 _runtime‚14394.056924154 /loss/FRt2/detection_loss‚1.1559267320577715 1loss/SRqqt2/detection_loss‚0.5871896181928071 /loss/FBt2/detection_loss‚0.5307965972186888 epoch‚42 _step‚1397ñÉBž4Ž 3loss/SRqqt1/assignment_loss‚0.36316494998515764 2loss/SRqqt2/assignment_loss‚0.4358955210990837 3loss/FBt1/assignment_loss‚9.182144265450023e-05 /loss/FRt1/detection_loss‚0.5218292850963414 0loss/FRt1/assignment_loss‚1.4312736665681922 0loss/SRqqt1/detection_loss‚0.963595663408204 trainer/global_step‚16949 % _timestamp‚1.7842872019315906e+09 _runtime‚14394.056924154 /loss/FRt2/detection_loss‚1.1559267320577715 1loss/SRqqt2/detection_loss‚0.5871896181928071 /loss/FBt2/detection_loss‚0.5307965972186888 epoch‚42 _step‚1397 0loss/FRt2/assignment_loss‚0.7878845096506192 2loss/FBt2/assignment_loss‚0.002465879553582795 0loss/FBt1/detection_loss‚0.37839067656823316 %loss/total_loss‚7.158504920841336õ ÖéAg“Ÿ4‚(:ˆ çŸèÒ£…Ó¢ gpu.0.gpu‚51.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚¯F 4‚(:’ çŸèÒã”Ǥ' disk.nvme0c0n1.out‚55505.0048828125 disk.sda.in‚ 0.0078125 disk.sda.out‚ 595.046875# memory_percent‚97.7046251296997" proc.memory.rssMB‚ 2772.3828125* proc.memory.percent‚16.921281814575195 cpu‚26.010966052908834 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144962.07421875" disk.dm-0.out‚55505.0048828125 network.sent‚ 2.6959324e+07 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 595.046875 network.recv‚ 3.813815e+06# disk.nvme0n1.in‚144962.07421875! disk.rbd1.out‚119.88720703125 disk.rbd3.out‚0% disk.nvme0n1.out‚55505.0048828125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚144962.07421875 disk.sda2.in‚0* disk./.usagePercent‚35.989670495365154$ disk./.usageGB‚536.2720985412598 disk.sda2.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 376.07421875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.sda3.out‚ 595.046875h¨“z“¡4‚(:ˆ öŸèÒÚ gpu.0.gpu‚62.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¥Y¢4‚(:‘ öŸèÒž›ß¤ network.sent‚ 2.7022367e+07 network.recv‚ 3.818025e+06# disk./.usageGB‚536.270694732666% disk.nvme0c0n1.in‚144962.07421875 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625# disk.nvme0n1.in‚144962.07421875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2772.3828125 proc.cpu.threads‚17 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.rbd3.out‚0 disk.dm-0.in‚144962.07421875 disk.sda1.in‚0* proc.memory.percent‚16.921281814575195 cpu‚26.010677353580892) disk./.usagePercent‚35.98957628459256 disk.dm-1.out‚ 595.23828125 disk.rbd0.in‚0 disk.sda.out‚ 595.23828125 disk.sda2.out‚0 disk.sda3.out‚ 595.23828125! disk.rbd1.out‚119.88720703125 disk.rbd3.in‚0# disk.nvme0n1.out‚55518.76171875% disk.nvme0c0n1.out‚55518.76171875 disk.drbd2165.in‚0$ memory_percent‚97.70834445953369( proc.memory.availableMB‚ 375.46484375 disk.sda2.in‚0 disk.dm-0.out‚55518.76171875 disk.sda1.out‚0 disk.drbd2165.out‚0?ó"[“£4‚(:ˆ … èÒÙý¡¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–pè ¤4‚(:• … èÒ’·Î¤ disk.drbd2165.in‚0 disk.drbd2165.out‚0! disk.rbd1.out‚119.94970703125 network.sent‚ 2.7026487e+07) disk./.usagePercent‚35.98998589664734 disk.sda1.in‚0 disk.sda.out‚ 595.44140625 disk.sda2.in‚0 disk.sda3.in‚ 0.0078125# memory_percent‚97.7053165435791 network.recv‚ 3.820567e+06# disk./.usageGB‚536.276798248291 disk.rbd0.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17# disk.nvme0n1.in‚144962.07421875 disk.rbd1.in‚0.0625 disk.dm-0.in‚144962.07421875 disk.dm-1.out‚ 595.44140625" proc.memory.rssMB‚ 2772.3828125 disk.sda3.out‚ 595.44140625 disk.dm-1.in‚ 0.0078125% disk.nvme0n1.out‚55532.5810546875% disk.nvme0c0n1.in‚144962.07421875 disk.sda.in‚ 0.0078125' proc.memory.availableMB‚ 375.9609375 disk.sda1.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚55532.5810546875 disk.sda2.out‚0 disk.rbd3.out‚0" disk.dm-0.out‚55532.5810546875* proc.memory.percent‚16.921281814575195 cpu‚26.010424996834757@ë}O4¥4/ _wandbruntime‚14433  _runtime‚14433±—9©É¦4à & lr-AdamW/pg1‚2.681797571217542e-05 & lr-AdamW/pg2‚2.681797571217542e-05 trainer/global_step‚16999 $ _timestamp‚1.784287241077476e+09 _runtime‚14433.203045944 _step‚1398-¡ïDΧ4È _runtime‚14433.203045944 _step‚1398 & lr-AdamW/pg1‚2.681797571217542e-05 & lr-AdamW/pg2‚2.681797571217542e-05 trainer/global_step‚16999 $ _timestamp‚1.784287241077476e+09ö ÀYü4¨4/ _wandbruntime‚14433  _runtime‚14433g/‘o(©4€P WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157464064 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2584175144042496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292279896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17982197605888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36588995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚569081182046208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17954765198895 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17981856266752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157464064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587260215855104 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2570880543592448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2584175127003136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66365716 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2584175127003136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134815832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157474041 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17954424649005 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17981856266752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292289875 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66365438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29776718 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2602156983269888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29776721 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚569081163433984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _step‚1399 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2602157341648384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2570880560631808 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2602157341648384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2¶¢sjœª4…P UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚569081182046208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17954765198895 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17981856266752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157464064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587260215855104 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2570880543592448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2584175127003136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66365716 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2584175127003136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134815832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157474041 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17954424649005 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17981856266752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292289875 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66365438 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29776718 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2602156983269888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29776721 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚569081163433984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _step‚1399 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2602157341648384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2570880560631808 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2602157341648384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157464064 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2584175144042496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292279896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17982197605888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36588995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0÷ 4K*y4«4/ _wandbruntime‚14433  _runtime‚14433ì€ãH'¬4ŠN OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134823466 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66369504 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17983210240000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17982868893696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚16999 _step‚1400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292306573 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2571027054264320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29778463 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17955776252729 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66369232 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2571027071303680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2588982847556409 $ _timestamp‚1.784287241689416e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157473116 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292306573 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17955435702727 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157483107 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134823464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587294666742272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17983210240000 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2602305236926464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚587294771827200 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2584322385072128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17982868893696 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157483107 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2584322385072128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ϵ›}'­4N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292306573 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17955435702727 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157483107 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134823464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587294666742272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17983210240000 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2602305236926464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚587294771827200 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2584322385072128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17982868893696 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157483107 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2584322385072128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134823466 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66369504 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17983210240000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17982868893696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚16999 _step‚1400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292306573 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2571027054264320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29778463 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17955776252729 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66369232 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2571027071303680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2588982847556409 $ _timestamp‚1.784287241689416e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157473116 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169ø iz -4®4/ _wandbruntime‚14433  _runtime‚14433!Ó ¯4ˆ 0loss/FRt2/assignment_loss‚0.7859026434849914 2loss/FBt1/assignment_loss‚0.004262741578377332 /loss/FRt1/detection_loss‚0.5097282387877135 %loss/total_loss‚7.066959716022256 epoch‚42 $ _timestamp‚1.784287241692776e+09 _runtime‚14433.818125614 /loss/FRt1/assignment_loss‚1.508561650105253 2loss/SRqqt1/assignment_loss‚0.3439491237333875 /loss/FRt2/detection_loss‚1.2453778304405572 /loss/FBt2/detection_loss‚0.4582315280883839 trainer/global_step‚16999 3loss/SRqqt2/assignment_loss‚0.43386314172629076 4loss/FBt2/assignment_loss‚2.6698723429571406e-05 1loss/SRqqt2/detection_loss‚0.6063521860504018 /loss/FBt1/detection_loss‚0.3788672395677313 1loss/SRqqt1/detection_loss‚0.7918366937357394 _step‚1401²§·­°4 1loss/SRqqt1/detection_loss‚0.7918366937357394 _step‚1401 0loss/FRt2/assignment_loss‚0.7859026434849914 2loss/FBt1/assignment_loss‚0.004262741578377332 /loss/FRt1/detection_loss‚0.5097282387877135 %loss/total_loss‚7.066959716022256 epoch‚42 $ _timestamp‚1.784287241692776e+09 _runtime‚14433.818125614 /loss/FRt1/assignment_loss‚1.508561650105253 2loss/SRqqt1/assignment_loss‚0.3439491237333875 /loss/FRt2/detection_loss‚1.2453778304405572 /loss/FBt2/detection_loss‚0.4582315280883839 trainer/global_step‚16999 3loss/SRqqt2/assignment_loss‚0.43386314172629076 4loss/FBt2/assignment_loss‚2.6698723429571406e-05 1loss/SRqqt2/detection_loss‚0.6063521860504018 /loss/FBt1/detection_loss‚0.3788672395677313ù W½í–“±4‚(:ˆ ” èÒÍÆþ£ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚267.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç†ß ²4‚(:• ” èÒú¼¥ disk.dm-0.in‚144962.07421875 disk.sda1.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144962.07421875 disk.sda2.out‚0$ disk./.usageGB‚536.2899703979492" proc.memory.rssMB‚ 2772.3828125) disk./.usagePercent‚35.99086989066308& disk.nvme0n1.out‚55545.34326171875 disk.sda1.out‚0 disk.dm-1.in‚ 0.0078125 cpu‚26.01008315093659 disk.sda2.in‚0 disk.sda3.out‚ 595.6953125 disk.rbd1.in‚0.0625! disk.rbd1.out‚119.94970703125# disk.dm-0.out‚55545.34326171875 disk.sda3.in‚ 0.0078125& proc.memory.availableMB‚ 376.109375* proc.memory.percent‚16.921281814575195 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 595.6953125 network.recv‚ 3.823784e+06 disk.rbd3.in‚0# disk.nvme0n1.in‚144962.07421875 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.dm-1.out‚ 595.6953125 disk.rbd3.out‚0( disk.nvme0c0n1.out‚55545.34326171875$ memory_percent‚97.70441055297852 network.sent‚ 2.7031568e+07*C±“³4‚(:ˆ £ èҀ幢 gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚238.946' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.7892 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=R ë"´4‚(:— £ èÒ¡§ˆ¥ disk.dm-1.out‚ 595.93359375 disk.dm-0.in‚144962.07421875# disk.nvme0n1.in‚144962.07421875" proc.memory.rssMB‚ 2772.3828125* proc.memory.percent‚16.921281814575195 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0# disk.dm-0.out‚55554.80712890625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625! disk.rbd1.out‚120.01220703125 cpu‚26.00978129058141 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 595.93359375& disk.nvme0n1.out‚55554.80712890625# memory_percent‚97.7055549621582$ disk./.usageGB‚536.2957992553711 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125& proc.memory.availableMB‚ 375.921875 network.sent‚ 2.7096738e+07) disk./.usagePercent‚35.99126107017539% disk.nvme0c0n1.in‚144962.07421875 disk.sda.out‚ 595.93359375 network.recv‚ 3.828241e+06( disk.nvme0c0n1.out‚55554.80712890625 disk.dm-1.in‚ 0.0078125¿¼Db4µ4/ _wandbruntime‚14473  _runtime‚14473pû˶4Å trainer/global_step‚17049 $ _timestamp‚1.784287281032081e+09 _runtime‚14473.157698619 _step‚1402 ' lr-AdamW/pg1‚2.5948376194686377e-05 ' lr-AdamW/pg2‚2.5948376194686377e-05 ÀŽÒз4Ê ' lr-AdamW/pg1‚2.5948376194686377e-05 ' lr-AdamW/pg2‚2.5948376194686377e-05 trainer/global_step‚17049 $ _timestamp‚1.784287281032081e+09 _runtime‚14473.157698619 _step‚1402ú ?!Ñ4¸4/ _wandbruntime‚14473  _runtime‚14473L÷PÞ¹4P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29863971 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570752601360384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2578206094225408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚14473.158952025 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36693031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1403 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135197432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2609570088631808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36693277 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚17049 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570752619972608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18230454178816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588983158534144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135197434 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157918516 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2609569730253312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18005041135913 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2591537195524096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66557248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29863968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2609569730253312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135197432 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2578206077186048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157918516 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18032893107712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 % _timestamp‚1.7842872810331793e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2591537195524096 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872¦E–* (º4†P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29863971 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570752601360384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2578206094225408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚14473.158952025 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36693031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1403 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135197432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2609570088631808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36693277 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚17049 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570752619972608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18230454178816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588983158534144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135197434 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157918516 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2609569730253312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18005041135913 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2591537195524096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66557248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29863968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2609569730253312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135197432 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2578206077186048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157918516 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18032893107712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 % _timestamp‚1.7842872810331793e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2591537195524096 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976û ËT4»4/ _wandbruntime‚14473  _runtime‚14473ìh²'¼4ŠN D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135205064 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293132632 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36695118 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2609718342295552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18006392739637 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2596358640047555 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2578352604897280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18033905741824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18033564395520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚17049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157937559 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157927568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66561095 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18033564395520 _runtime‚14473.77092352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18033905741824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1404 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293142625 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2591684436553728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚570786030118912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18231485719040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66560831 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293142625 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2609718342295552 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2596358997636917 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135205066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29865716 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157937559 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2591684436553728 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2591684419514368 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18231567997440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135205066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36695379 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2578352587857920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293132632 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157927568 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2609717983909888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18006052189635 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29865713 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589017515837952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚589017616728576`Ó3Ž½4N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18033905741824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚1404 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293142625 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2591684436553728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚570786030118912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18231485719040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66560831 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293142625 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2609718342295552 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2596358997636917 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135205066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29865716 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157937559 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2591684436553728 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2591684419514368 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18231567997440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135205066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36695379 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2578352587857920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeÊX£ñ‡viceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293132632 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157927568 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2609717983909888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18006052189635 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29865713 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589017515837952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚589017616728576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135205064 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293132632 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36695118 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2609718342295552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18006392739637 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2596358640047555 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2578352604897280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18033905741824 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18033564395520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚17049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157937559 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157927568 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66561095 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18033564395520 _runtime‚14473.77092352ü –-44¾4/ _wandbruntime‚14473  _runtime‚14473Ø›¿4‰ %loss/total_loss‚6.727413281469021 _runtime‚14473.773680675 0loss/FRt1/assignment_loss‚1.4941657087656373 3loss/SRqqt2/assignment_loss‚0.44952016390365007 3loss/FBt2/assignment_loss‚0.0017246285132570855 /loss/FRt1/detection_loss‚0.4835025267048059 trainer/global_step‚17049 % _timestamp‚1.7842872816483397e+09 1loss/SRqqt2/detection_loss‚0.5689097624648876 /loss/FBt2/detection_loss‚0.3188086545976278 epoch‚42 0loss/FRt2/assignment_loss‚0.7402329285364686 3loss/SRqqt1/assignment_loss‚0.33318401936606135 /loss/FRt2/detection_loss‚1.1898766181611544 1loss/SRqqt1/detection_loss‚0.8430713594366728 _step‚1405 2loss/FBt1/assignment_loss‚0.009678419109812882 .loss/FBt1/detection_loss‚0.294738491908985훑ôÀ4Ž 2loss/FBt1/assignment_loss‚0.009678419109812882 .loss/FBt1/detection_loss‚0.294738491908985 %loss/total_loss‚6.727413281469021 _runtime‚14473.773680675 0loss/FRt1/assignment_loss‚1.4941657087656373 3loss/SRqqt2/assignment_loss‚0.44952016390365007 3loss/FBt2/assignment_loss‚0.0017246285132570855 /loss/FRt1/detection_loss‚0.4835025267048059 trainer/global_step‚17049 % _timestamp‚1.7842872816483397e+09 1loss/SRqqt2/detection_loss‚0.5689097624648876 /loss/FBt2/detection_loss‚0.3188086545976278 epoch‚42 0loss/FRt2/assignment_loss‚0.7402329285364686 3loss/SRqqt1/assignment_loss‚0.33318401936606135 /loss/FRt2/detection_loss‚1.1898766181611544 1loss/SRqqt1/detection_loss‚0.8430713594366728 _step‚1405ý Ê=Õ^›Á4‚(: ² èÒÀ×£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚279.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.867999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SõgGÂ4‚(:† ² èÒÆø¤¥ disk.sda3.in‚ 0.0078125 disk.sda2.out‚0 disk.dm-1.out‚596.125* proc.memory.percent‚16.921281814575195& disk.nvme0n1.out‚55568.26220703125 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.709966e+07$ disk./.usageGB‚536.3024826049805 disk.dm-1.in‚ 0.0078125" proc.memory.rssMB‚ 2772.3828125 cpu‚26.00949487267366 disk.rbd1.in‚0.0625 disk.dm-0.in‚144962.07421875# disk.dm-0.out‚55568.26220703125# disk.nvme0n1.in‚144962.07421875 disk.drbd2165.out‚0 disk.sda.out‚596.125$ proc.memory.availableMB‚375.6875% disk.nvme0c0n1.in‚144962.07421875 disk.sda3.out‚596.125) disk./.usagePercent‚35.99170959537538! disk.rbd1.out‚120.01220703125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 network.recv‚ 3.831341e+06 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚97.70698547363281 disk.sda1.out‚0( disk.nvme0c0n1.out‚55568.26220703125,×TœÃ4‚(:‘ Á èÒ¹Õã gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷€Ã0!Ä4‚(:– Á èÒ²›Û¤ disk.sda3.out‚ 596.3359375# disk.dm-0.out‚55576.89892578125 network.sent‚ 2.7163358e+07 network.recv‚ 3.83667e+06( disk.nvme0c0n1.out‚55576.89892578125# disk.nvme0n1.in‚144962.07421875 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.sda.out‚ 596.3359375 disk.rbd1.in‚0.0625$ memory_percent‚97.70770072937012" proc.memory.rssMB‚ 2772.3828125 proc.cpu.threads‚17! disk.rbd1.out‚120.07470703125 disk.rbd3.out‚0* proc.memory.percent‚16.921281814575195 disk.dm-1.in‚ 0.0078125* disk./.usagePercent‚35.992044453230164 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚536.3074722290039 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0078125 disk.dm-0.in‚144962.07421875& disk.nvme0n1.out‚55576.89892578125 disk.sda1.out‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 375.5703125 cpu‚26.00919484408176% disk.nvme0c0n1.in‚144962.07421875 disk.dm-1.out‚ 596.3359375ÃàÒˆžÅ4‚(:“ РèÒ•€Ê¢ gpu.0.gpu‚89.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚155.685' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.136999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù˱)Æ4‚(:‹ РèÒÊ䣦 disk.sda3.out‚ 596.65625 disk.dm-1.out‚ 596.65625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚144962.07421875% disk.nvme0c0n1.in‚144962.07421875 disk.sda.out‚ 596.65625 disk.sda3.in‚ 0.0078125! disk.rbd1.out‚120.07470703125 disk.sda2.in‚0 disk.sda1.out‚0$ proc.memory.availableMB‚375.5625* proc.memory.percent‚16.921281814575195 cpu‚26.008887427322037 proc.cpu.threads‚17 disk.sda2.out‚0# disk.nvme0n1.in‚144962.07421875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125) disk./.usagePercent‚35.99235012622605" disk.dm-0.out‚55588.3701171875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚55588.3701171875 disk.dm-1.in‚ 0.0078125% disk.nvme0n1.out‚55588.3701171875 disk.drbd2165.out‚0$ memory_percent‚97.70774841308594 network.recv‚ 3.847031e+06$ disk./.usageGB‚536.3120269775391 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2772.3828125 network.sent‚ 2.7171863e+07 disk.rbd1.in‚0.0625Ô–C4Ç4/ _wandbruntime‚14512  _runtime‚14512ЕùSÌÈ4Æ ' lr-AdamW/pg1‚2.5092289359546845e-05 ' lr-AdamW/pg2‚2.5092289359546845e-05 trainer/global_step‚17099 % _timestamp‚1.7842873207658002e+09 _runtime‚14512.891418856 _step‚1406 XïYÑÉ4Ë ' lr-AdamW/pg1‚2.5092289359546845e-05 ' lr-AdamW/pg2‚2.5092289359546845e-05 trainer/global_step‚17099 % _timestamp‚1.7842873207658002e+09 _runtime‚14512.891418856 _step‚1406þ T$&¡4Ê4/ _wandbruntime‚14512  _runtime‚14512A¬{(Ë4€P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7842873207670944e+09 _step‚1407 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158372968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2598899247005696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚572424057899008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2603587625991719 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135579032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293952000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135579034 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18055657622821 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135579034 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18281816805376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2598899247005696 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29951218 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2585531610779648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2616982477236736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590705874704384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18083588609536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2603587268402469 trainer/global_step‚17099 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158382945 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158372968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2585531627819008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158382945 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18083247270400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚572424039286784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18055998172711 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66748270 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18083588609536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2616982835615232 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2598899229966336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2598899229966336 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2616982477236736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚590705778000896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66748021 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088ç½ïU (Ì4…P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36796803 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135579032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293961979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18281738714112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18083247270400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158382945 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18083247270400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚572424039286784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18055998172711 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66748270 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18083588609536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2616982835615232 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2598899229966336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2598899229966336 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2616982477236736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚590705778000896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66748021 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7842873207670944e+09 _step‚1407 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158372968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2598899247005696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚572424057899008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2603587625991719 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135579032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293952000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135579034 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18055657622821 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135579034 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18281816805376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2598899247005696 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29951218 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2585531610779648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2616982477236736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590705874704384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18083588609536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2603587268402469 trainer/global_step‚17099 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158382945 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158372968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2585531627819008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2599046470995968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135586664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36799114 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293978677 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293968684 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2585678138490880 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2585678121451520 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2617131089278976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135586666 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚14513.503552045 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29952966 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293978677 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158392011 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚590740332105216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135586664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18084601243648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66751830 % _timestamp‚1.7842873213777704e+09 _step‚1408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2599046470995968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18084601243648 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158382020 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18084259897344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚17099 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135586666 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2603735147717425 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18056668676543 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚572457486657536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18282754780672<„‡0'Ï4N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18282845447680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2599046488035328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293978677 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158392011 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚590740332105216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135586664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18084601243648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66751830 % _timestamp‚1.7842873213777704e+09 _step‚1408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2599046470995968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18084601243648 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158382020 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18084259897344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚17099 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135586666 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2603735147717425 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18056668676543 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚572457486657536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18282754780672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2599046470995968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135586664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36799114 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293978677 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293968684 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2585678138490880 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2585678121451520 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2617131089278976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135586666 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚14513.503552045 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29952966 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158827420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30038471 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36901556 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2624395582598656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2606261281447936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2606261298487296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135960632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294798031 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158837397 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2624395582598656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 $ _timestamp‚1.784287360648693e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2610963418482977 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18133942772224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66939765 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚17149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158827420 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66940027 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18106614659619 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18134284111360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚574095477213184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2606261298487296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30038468 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2624395224220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574095495825408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18106274109729 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18333083043328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_byt3°ÂnÒ es.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158837397 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18133942772224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592428560256512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158827420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30038471 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36901556 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2624395582598656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2606261281447936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2606261298487296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135960632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294798031 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158837397 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2624395582598656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 $ _timestamp‚1.784287360648693e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2610963418482977 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18133942772224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66939765 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚17149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158827420 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66940027 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18106614659619 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18134284111360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚574095477213184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2606261298487296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30038468 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2624395224220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574095495825408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18106274109729 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18333083043328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158837397 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18133942772224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592428560256512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158836472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18107285163451 trainer/global_step‚17149 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36903585 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2624543477876736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2611111297797933 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294814729 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592463010070528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18334192668672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294804736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66943801 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2606408522477568 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30040213 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294804736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2606408522477568 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66943536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2624543836262400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135968264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842873612626128e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158846463 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2606408539516928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1412 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36903323 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294814729 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2624543836262400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592463117252608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18107625713453 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158836472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2593003655045120ñg°¼%á4N D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842873612626128e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158846463 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2606408539516928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1412 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36903323 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294814729 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2624543836262400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592463117252608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18107625713453 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158836472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2593003655045120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18135296745472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18334104098816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135968266 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574128905971712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2593003672084480 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158836472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18107285163451 trainer/global_step‚17149 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36903585 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2624543477876736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2611111297797933 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294814729 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592463010070528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18334192668672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294804736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66943801 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2606408522477568 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30040213 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294804736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2606408522477568 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66943536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2624543836262400 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135968264 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 Q†•Ìê4Æ _runtime‚14592.722667715 _step‚1414 ' lr-AdamW/pg1‚2.3420872804320957e-05 ' lr-AdamW/pg2‚2.3420872804320957e-05 trainer/global_step‚17199 % _timestamp‚1.7842874005970578e+09'pÿÑë4Ë _step‚1414 ' lr-AdamW/pg1‚2.3420872804320957e-05 ' lr-AdamW/pg2‚2.3420872804320957e-05 trainer/global_step‚17199 % _timestamp‚1.7842874005970578e+09 _runtime‚14592.722667715† ¨A*4ì4/ _wandbruntime‚14592  _runtime‚14592 Éþ(í4P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2613623349968896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚594151433062912 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18157231146527 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159281872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2618339568563485 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2613623332929536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2631807971203584 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2613623332929536 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136342232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295634083 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚594151323776512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2618339926152735 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159281872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚575766933751808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295624104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136342232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2631807971203584 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18184638274048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159291849 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18184979613184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18384408636928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2631808329582080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18384499311104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18156890596637 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136342234 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18184979613184 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2600182695006208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575766915139584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136342234 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67131590 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159291849 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842874005981503e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67131319 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37005869 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚14592.723891937 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30125718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2613623349968896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37005601 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2631808329582080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18184638274048 _step‚1415&gˆì (î4†P `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18384408636928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2631808329582080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18384499311104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18156890596637 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136342234 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18184979613184 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2600182695006208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575766915139584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136342234 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67131590 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159291849 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842874005981503e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67131319 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37005869 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚14592.723891937 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30125718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2613623349968896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37005601 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2631808329582080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18184638274048 _step‚1415 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2613623349968896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚594151433062912 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18157231146527 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159281872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2618339568563485 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2613623332929536 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2631807971203584 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2613623332929536 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136342232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295634083 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚594151323776512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2618339926152735 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159281872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚575766933751808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295624104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136342232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2631807971203584 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18184638274048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159291849 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18184979613184‡ ÿ ®G4ï4/ _wandbruntime‚14593  _runtime‚14593A·ç$ð4ŠN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2613770590998528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2631956224860160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37007953 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136349864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295650781 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2631956583245824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30127463 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚17199 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67135139 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚1416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18185650900992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2618487090289079 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2618487447878441 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295640788 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2600329205678080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.løxæì arge_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159300915 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136349866 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18157901650359 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _runtime‚14593.33410752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136349866 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18385517508608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295650781 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2613770573959168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18185650900992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159290924 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159290924 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18385420550144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842874012083309e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2613770590998528 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136349864 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295650781 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2631956583245824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30127463 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚17199 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67135139 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚1416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18185650900992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2618487090289079 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2618487447878441 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295640788 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2600329205678080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159300915 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136349866 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18157901650359 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _runtime‚14593.33410752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136349866 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18385517508608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295650781 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2613770573959168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18185650900992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159290924 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159290924 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18385420550144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7842874012083309e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2613770590998528 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2613770590998528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2631956224860160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37007953 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77ˆ „º?á4ò4/ _wandbruntime‚14593  _runtime‚14593ån„á ó4‡ 1loss/SRqqt2/detection_loss‚0.6443933780243158 /loss/FBt2/detection_loss‚0.5856141242959645 trainer/global_step‚17199 0loss/FRt1/assignment_loss‚1.5230375043166582 /loss/FRt1/detection_loss‚0.5253878675839567 /loss/FBt1/detection_loss‚0.3380938134361344 % _timestamp‚1.7842874012123742e+09 _runtime‚14593.337702146 %loss/total_loss‚7.340445161743327 2loss/SRqqt2/assignment_loss‚0.4177292594602273 1loss/SRqqt1/detection_loss‚0.8175413953719957 1loss/FBt2/assignment_loss‚0.03405913198670841 epoch‚42 _step‚1417 0loss/FRt2/assignment_loss‚0.8068422315212397 2loss/SRqqt1/assignment_loss‚0.3704552820045574 3loss/FBt1/assignment_loss‚0.0008553783134927568 /loss/FRt2/detection_loss‚1.2764357954280758Ø^?—ô4Œ %loss/total_loss‚7.340445161743327 2loss/SRqqt2/assignment_loss‚0.4177292594602273 1loss/SRqqt1/detection_loss‚0.8175413953719957 1loss/FBt2/assignment_loss‚0.03405913198670841 epoch‚42 _step‚1417 0loss/FRt2/assignment_loss‚0.8068422315212397 2loss/SRqqt1/assignment_loss‚0.3704552820045574 3loss/FBt1/assignment_loss‚0.0008553783134927568 /loss/FRt2/detection_loss‚1.2764357954280758 1loss/SRqqt2/detection_loss‚0.6443933780243158 /loss/FBt2/detection_loss‚0.5856141242959645 trainer/global_step‚17199 0loss/FRt1/assignment_loss‚1.5230375043166582 /loss/FRt1/detection_loss‚0.5253878675839567 /loss/FBt1/detection_loss‚0.3380938134361344 % _timestamp‚1.7842874012123742e+09 _runtime‚14593.337702146‰ còØ?‘õ4‚(:† ª¡èÒöŠ¢ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§•ƒK!ö4‚(:– ª¡èÒ¦ÒÈ¥ disk.sda3.out‚ 598.06640625 disk.sda.out‚ 598.06640625 disk.dm-1.out‚ 598.06640625 disk.sda1.out‚0* proc.memory.percent‚16.941428184509277 cpu‚26.00753771691462 disk.sda3.in‚ 0.0078125 disk.dm-0.in‚144962.07421875# disk.dm-0.out‚55650.51904296875 disk.drbd2165.out‚0) disk./.usagePercent‚35.99464113764997 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625# disk.nvme0n1.in‚144962.07421875 disk.rbd3.out‚0% disk.nvme0c0n1.in‚144962.07421875 network.recv‚ 3.869609e+06 disk.sda2.in‚0 disk.sda2.out‚0! disk.rbd1.out‚120.32470703125& disk.nvme0n1.out‚55650.51904296875 disk.drbd2165.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚55650.51904296875 network.sent‚ 2.7318354e+07 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125$ memory_percent‚97.72720336914062# proc.memory.availableMB‚372.375# proc.memory.rssMB‚ 2775.68359375 proc.cpu.threads‚17$ disk./.usageGB‚536.3461647033691 disk.rbd3.in‚0tBn¢œ÷4‚(:‘ ¹¡èÒ×­Ô¢ gpu.0.gpu‚15.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªf¾lø4‚(:” ¹¡èÒìý⤠disk.dm-1.out‚ 600.05078125 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.sda2.out‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 356.11328125 proc.memory.rssMB‚ 2791.8125 disk.sda2.in‚0 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 600.05078125% disk.nvme0c0n1.in‚144962.07421875* proc.memory.percent‚17.039871215820312 proc.cpu.threads‚17$ disk./.usageGB‚536.3593215942383 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.drbd2165.in‚0) disk./.usagePercent‚35.99552410763556 disk.dm-0.in‚144962.07421875" disk.dm-0.out‚55678.2998046875# disk.nvme0n1.in‚144962.07421875 disk.sda1.out‚0 disk.rbd0.in‚0# memory_percent‚97.8264570236206 cpu‚26.011200452959923 network.recv‚ 3.873977e+06! disk.rbd1.out‚120.38720703125 network.sent‚ 2.7381413e+07 disk.sda.out‚ 600.05078125% disk.nvme0n1.out‚55678.2998046875' disk.nvme0c0n1.out‚55678.2998046875˜Óad4ù4/ _wandbruntime‚14610  _runtime‚14610"ÛUÒ¾9ú4ˆ{ 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.578766942024231 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24078303575515747 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27567896246910095 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03602704405784607 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7931326627731323 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7993287444114685 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9889349341392517 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6020534038543701 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8706626892089844 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3605774939060211 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23729991912841797 particle/accuracy_3_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6093519926071167 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08775735646486282 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9966942071914673 +jet/accuracy_3_of_4‚0.07847467809915543 particle/accuracy_1_of_1‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2658259868621826 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29943642020225525 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.627116858959198 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.697257936000824 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20004983246326447 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33313310146331787 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 *jet/accuracy_2_of_3‚0.5714499354362488 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5639784932136536 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3228549659252167 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5216615200042725 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7661858797073364 6validation_average_jet_accuracy‚0.3750036656856537 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2768711447715759 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7216867804527283 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +particle/sensitivity‚0.6155973076820374 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02211873233318329 )jet/accuracy_2_of_4‚0.802596390247345 /particle/accuracy_4_of_4‚0.9555904865264893 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07754897326231003 particle/accuracy_2_of_2‚1 epoch‚42 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.575536847114563 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8477446436882019 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6019688844680786 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022339852526783943 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7642570734024048 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3318648338317871 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.257768452167511 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3606017827987671 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6685312986373901 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5311490893363953 *jet/accuracy_1_of_2‚0.7932858467102051 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950399994850159 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886239171028137 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6238657832145691 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2680591344833374 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/specificity‚0.9028616547584534 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2391817718744278 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20783312618732452 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2005717158317566 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28410962224006653 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5948527455329895 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021770862862467766 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.773265540599823 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5270238518714905 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1588977724313736 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7263038754463196 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5424041748046875 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2386927604675293 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_3_of_3‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6093519926071167 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6662253737449646 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7157047390937805 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6629440784454346 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_5_of_5‚0 _step‚1418 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02248334512114525 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7219009399414062 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857453107833862 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7260959148406982 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6705397367477417 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9941893219947815 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6355202794075012 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6196711659431458 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960261583328247 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7219009399414062 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6336262822151184 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2548772394657135 _runtime‚14610.473874737 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6304256319999695 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.617529034614563 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2501316964626312 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22248463332653046 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6339648962020874 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043357640504837036 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6160141825675964 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021617569029331207 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2704125940799713 (particle/accuracy‚0.8411690592765808 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 particle/accuracy_4_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04078985005617142 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04600117728114128 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6649332642555237 'particle/f_score‚0.6246917247772217 )loss/val_total_loss‚8.047289585405357 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6080088019371033 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.604535698890686 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6155632138252258 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8516414165496826 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22586461901664734 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9883613586425781 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25149035453796387 particle/accuracy_1_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9254838824272156 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8966553211212158 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972149133682251 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6632998585700989 ,jet/accuracy_4_of_4‚0.008297258988022804 % _timestamp‚1.7842874183474264e+09 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 *jet/accuracy_1_of_4‚0.9816179871559143 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9121111631393433 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3720945119857788 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2148604393005371 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20647525787353516 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6625979542732239 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27583131194114685 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24180053174495697 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6492030620574951 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2088395357131958 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8706626892089844 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416666626930237 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458134651184 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26651084423065186 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7252529859542847 particle/accuracy_5_of_5‚1 trainer/global_step‚17199 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7658265233039856 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.700886070728302 >&Purity/1FRtjNèçÐ0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7213152050971985 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_3‚0.05562012642621994 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4946373999118805 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44486090540885925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2658259868621826 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02211873233318329rݘ“=û4{ 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857453107833862 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7260959148406982 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6705397367477417 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9941893219947815 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6355202794075012 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6196711659431458 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960261583328247 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7219009399414062 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6336262822151184 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2548772394657135 _runtime‚14610.473874737 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6304256319999695 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.617529034614563 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2501316964626312 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22248463332653046 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6339648962020874 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043357640504837036 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6160141825675964 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021617569029331207 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2704125940799713 (particle/accuracy‚0.8411690592765808 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 particle/accuracy_4_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04078985005617142 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04600117728114128 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6649332642555237 'particle/f_score‚0.6246917247772217 )loss/val_total_loss‚8.047289585405357 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6080088019371033 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.604535698890686 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6155632138252258 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8516414165496826 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22586461901664734 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9883613586425781 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25149035453796387 particle/accuracy_1_of_4‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9254838824272156 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8966553211212158 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972149133682251 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6632998585700989 ,jet/accuracy_4_of_4‚0.008297258988022804 % _timestamp‚1.7842874183474264e+09 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 *jet/accuracy_1_of_4‚0.9816179871559143 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9121111631393433 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3720945119857788 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2148604393005371 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20647525787353516 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6625979542732239 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27583131194114685 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24180053174495697 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6492030620574951 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2088395357131958 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8706626892089844 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416666626930237 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458134651184 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26651084423065186 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7252529859542847 particle/accuracy_5_of_5‚1 trainer/global_step‚17199 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7658265233039856 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.700886070728302 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7213152050971985 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_3‚0.05562012642621994 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4946373999118805 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44486090540885925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2658259868621826 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02211873233318329 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.578766942024231 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24078303575515747 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27567896246910095 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03602704405784607 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7931326627731323 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7993287444114685 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9889349341392517 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6020534038543701 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8706626892089844 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3605774939060211 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23729991912841797 particle/accuracy_3_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39120373129844666 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6093519926071167 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08775735646486282 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9966942071914673 +jet/accuracy_3_of_4‚0.07847467809915543 particle/accuracy_1_of_1‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2658259868621826 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29943642020225525 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.627116858959198 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.697257936000824 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20004983246326447 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33313310146331787 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 *jet/accuracy_2_of_3‚0.5714499354362488 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5639784932136536 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3228549659252167 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5216615200042725 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7661858797073364 6validation_average_jet_accuracy‚0.3750036656856537 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2768711447715759 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7216867804527283 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +particle/sensitivity‚0.6155973076820374 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02211873233318329 )jet/accuracy_2_of_4‚0.802596390247345 /particle/accuracy_4_of_4‚0.9555904865264893 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07754897326231003 particle/accuracy_2_of_2‚1 epoch‚42 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.575536847114563 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8477446436882019 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6019688844680786 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022339852526783943 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7642570734024048 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3318648338317871 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.257768452167511 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3606017827987671 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6685312986373901 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5311490893363953 *jet/accuracy_1_of_2‚0.7932858467102051 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950399994850159 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886239171028137 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6238657832145691 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2680591344833374 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/specificity‚0.9028616547584534 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2391817718744278 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20783312618732452 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2005717158317566 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28410962224006653 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5948527455329895 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.021770862862467766 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.773265540599823 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5270238518714905 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1588977724313736 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7263038754463196 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5424041748046875 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2386927604675293 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_3_of_3‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6093519926071167 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6662253737449646 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7157047390937805 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6629440784454346 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_5_of_5‚0 _step‚1418 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02248334512114525 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7219009399414062Š hÞ´‘žü4‚(:“ È¡èÒ¨ßÍ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.469' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.693799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅêP!ý4‚(:– È¡èÒÊâ¤% disk.nvme0c0n1.in‚144962.07421875 disk.rbd3.out‚0" disk.dm-0.out‚55703.4345703125# disk.nvme0n1.in‚144962.07421875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚536.3624076843262 disk.drbd2165.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 356.08984375 disk.rbd0.out‚0! disk.rbd1.out‚120.38720703125* proc.memory.percent‚17.039871215820312* disk./.usagePercent‚35.995731217730764 disk.rbd3.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚55703.4345703125 disk.sda3.in‚ 0.0078125 disk.dm-0.in‚144962.07421875 disk.sda1.out‚0$ memory_percent‚97.82660007476807 proc.memory.rssMB‚ 2791.8125 network.sent‚ 2.7386453e+07 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125% disk.nvme0n1.out‚55703.4345703125 disk.sda.out‚ 602.02734375 disk.sda3.out‚ 602.02734375 cpu‚26.011171966155306 network.recv‚ 3.877365e+06 disk.drbd2165.in‚0 disk.dm-1.out‚ 602.02734375 disk.rbd1.in‚0.0625)Lö8“þ4‚(:ˆ סèÒò•Á¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.992' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“M…ÿ4‚(:” סèÒßö¤ disk.sda3.out‚ 602.80859375$ memory_percent‚97.82624244689941' proc.memory.availableMB‚ 356.1484375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.out‚ 602.80859375 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 cpu‚26.010853567651342 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd1.out‚120.44970703125 disk.rbd3.out‚0" disk.dm-0.out‚55716.5068359375 disk.drbd2165.out‚0 disk.sda.in‚ 0.0078125 proc.memory.rssMB‚ 2791.8125$ disk./.usageGB‚536.3690414428711 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 network.recv‚ 3.881537e+06 disk.sda1.out‚0* proc.memory.percent‚17.039871215820312 disk.dm-0.in‚144962.07421875 network.sent‚ 2.7444019e+07) disk./.usagePercent‚35.99617641483281% disk.nvme0c0n1.in‚144962.07421875' disk.nvme0c0n1.out‚55716.5068359375 disk.sda2.in‚0 disk.dm-1.out‚ 602.80859375# disk.nvme0n1.in‚144962.07421875% disk.nvme0n1.out‚55716.5068359375ŽêQ4€5/ _wandbruntime‚14649  _runtime‚14649Ö ’˜Ê5Ä _runtime‚14649.815033238 _step‚1419 & lr-AdamW/pg1‚2.260565043619675e-05 & lr-AdamW/pg2‚2.260565043619675e-05 trainer/global_step‚17249 % _timestamp‚1.7842874576894703e+09ŠÙýäÏ‚5É % _timestamp‚1.7842874576894703e+09 _runtime‚14649.815033238 _step‚1419 & lr-AdamW/pg1‚2.260565043619675e-05 & lr-AdamW/pg2‚2.260565043619675e-05 trainer/global_step‚17249‹ ûœØÔ4ƒ5/ _wandbruntime‚14649  _runtime‚14649p† („5€P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18253066981376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2609416891813888 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136844500 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159816623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577454265165824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136844498 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18453825713664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2627642459172552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18225226808774 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2641168018042880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37139654 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚17249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136844498 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2641168018042880 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2622914951061504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18253408320512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67355410 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595908002302464 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2609416874774528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296651144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2627642101583302 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2622914951061504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18453737136640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159806646 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2622914968100864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595908109491712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18253408320512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30215488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18253408320512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30215488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2641168018042880 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2622914951061504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18253408320512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67355410 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595908002302464 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2609416874774528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296651144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2627642101583302 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2622914951061504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18453737136640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159806646 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2622914968100864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595908109491712 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296677821 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577487712536576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577487693924352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18254079608320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2623062192091136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30217233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18226578412498 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2641316630085120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136852132 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2627789623308896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159815698 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67358937 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18226237862496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136852132 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136852130 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159825689 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67359197 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296667828 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2641316630085120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚1421 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136852130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2623062209130496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18254420954624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37141961“Ýs'ˆ5N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18254079608320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2623062192091136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30217233 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18226578412498 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2641316630085120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136852132 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2627789623308896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159815698 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67358937 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18226237862496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136852132 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136852130 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159825689 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67359197 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296667828 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2641316630085120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚1421 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136852130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2623062209130496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18254420954624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37141961 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296677821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37141704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842874583014524e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595942558221312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296677821 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577487712536576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577487693924352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 Œ™Q±4‰5/ _wandbruntime‚14650  _runtime‚14650\ˆÑÿ Š5… _runtime‚14650.429425958 _step‚1422 0loss/FRt1/assignment_loss‚1.4252823899576674 2loss/FBt2/assignment_loss‚0.056466963228114835 /loss/FRt1/detection_loss‚0.5307628264615323 -loss/FRt2/detection_loss‚1.16488912982988 /loss/FBt1/detection_loss‚0.4377776492444972 % _timestamp‚1.7842874583040817e+09 2loss/SRqqt1/assignment_loss‚0.3112496299270735 /loss/FBt2/detection_loss‚0.3329222604070585 epoch‚43 trainer/global_step‚17249 0loss/FRt2/assignment_loss‚0.7476344976910245 1loss/SRqqt2/assignment_loss‚0.451773642194066 3loss/FBt1/assignment_loss‚0.0013532162733105885 1loss/SRqqt1/detection_loss‚0.8926542590538572 0loss/SRqqt2/detection_loss‚0.595515084486408 &loss/total_loss‚6.9482815487544904ñò4­‹5Š 2loss/SRqqt1/assignment_loss‚0.3112496299270735 /loss/FBt2/detection_loss‚0.3329222604070585 epoch‚43 trainer/global_step‚17249 0loss/FRt2/assignment_loss‚0.7476344976910245 1loss/SRqqt2/assignment_loss‚0.451773642194066 3loss/FBt1/assignment_loss‚0.0013532162733105885 1loss/SRqqt1/detection_loss‚0.8926542590538572 0loss/SRqqt2/detection_loss‚0.595515084486408 &loss/total_loss‚6.9482815487544904 _runtime‚14650.429425958 _step‚1422 0loss/FRt1/assignment_loss‚1.4252823899576674 2loss/FBt2/assignment_loss‚0.056466963228114835 /loss/FRt1/detection_loss‚0.5307628264615323 -loss/FRt2/detection_loss‚1.16488912982988 /loss/FBt1/detection_loss‚0.4377776492444972 % _timestamp‚1.7842874583040817e+09Ž ´Ð“Œ5‚(:ˆ æ¡èÒÃ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚266.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÇ5‚(:Ž æ¡èÒÒ°¦* proc.memory.percent‚17.039871215820312 network.recv‚ 3.884794e+06 disk.dm-0.in‚144962.07421875' disk.nvme0c0n1.out‚55736.0107421875 network.sent‚ 2.7449604e+07 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 603.015625 disk.sda2.in‚0 proc.memory.rssMB‚ 2791.8125 proc.cpu.threads‚17$ disk./.usageGB‚536.3754577636719 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0n1.in‚144962.07421875 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144962.07421875 disk.dm-1.out‚ 603.015625" disk.dm-0.out‚55736.0107421875( proc.memory.availableMB‚ 355.91796875( disk./.usagePercent‚35.9966070195054% disk.nvme0n1.out‚55736.0107421875 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625! disk.rbd1.out‚120.44970703125 disk.rbd3.in‚0 disk.dm-1.in‚ 0.0078125 disk.rbd3.out‚0$ memory_percent‚97.82764911651611 cpu‚26.010548010006296 disk.sda3.out‚ 603.015625@?ƒ^“Ž5‚(:ˆ õ¡èÒÛú㣠gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚234.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.9726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a/øG#5‚(:˜ õ¡èÒ‡ü¿¥# disk.nvme0n1.in‚144962.07421875 disk.rbd1.in‚0.0625 disk.drbd2165.in‚0* disk./.usagePercent‚35.996928052953336 disk.sda.out‚ 603.62890625 disk.sda3.in‚ 0.0078125& disk.nvme0n1.out‚55751.64013671875 network.recv‚ 3.889541e+06 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 603.62890625 network.sent‚ 2.7512272e+07# disk./.usageGB‚536.380241394043 disk.dm-1.out‚ 603.62890625 proc.memory.rssMB‚ 2791.8125! disk.rbd1.out‚120.57470703125 disk.dm-0.in‚144962.07421875 disk.sda.in‚ 0.0078125 cpu‚26.010251774588017 disk.sda2.in‚0# disk.dm-0.out‚55751.64013671875 disk.dm-1.in‚ 0.0078125 disk.rbd3.out‚0$ memory_percent‚97.82731533050537( proc.memory.availableMB‚ 355.97265625* proc.memory.percent‚17.039871215820312 disk.sda1.out‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144962.07421875( disk.nvme0c0n1.out‚55751.64013671875[ÛËèf5‚(:ˆ „¢èÒ…¿¢£ gpu.0.gpu‚51.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792ÄÌ›ô-785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 *}h‘5‚(: „¢èÒ…ÝŒ¥ disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.sda2.in‚0 disk.sda3.out‚ 603.80078125' proc.memory.availableMB‚ 356.0859375 cpu‚26.00997117917337# disk./.usageGB‚536.386791229248 disk.sda.out‚ 603.80078125 proc.memory.rssMB‚ 2791.8125 disk.dm-1.out‚ 603.80078125 disk.sda1.out‚0& disk.nvme0c0n1.out‚55768.001953125 disk.sda.in‚ 0.0078125# disk.nvme0n1.in‚144962.07421875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17! disk.rbd1.out‚120.57470703125 disk.dm-0.in‚144962.07421875 network.sent‚ 2.7517306e+07 network.recv‚ 3.892902e+06) disk./.usagePercent‚35.99736761788962% disk.nvme0c0n1.in‚144962.07421875 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚55768.001953125$ disk.nvme0n1.out‚55768.001953125$ memory_percent‚97.82662391662598* proc.memory.percent‚17.039871215820312 disk.rbd0.out‚0Ÿ”EB4’5/ _wandbruntime‚14689  _runtime‚14689à DÌ“5Æ trainer/global_step‚17299 % _timestamp‚1.7842874973949552e+09 _runtime‚14689.520566034 _step‚1423 ' lr-AdamW/pg1‚2.1804155447451033e-05 ' lr-AdamW/pg2‚2.1804155447451033e-05Ñ_’Ñ”5Ë ' lr-AdamW/pg2‚2.1804155447451033e-05 trainer/global_step‚17299 % _timestamp‚1.7842874973949552e+09 _runtime‚14689.520566034 _step‚1423 ' lr-AdamW/pg1‚2.1804155447451033e-05 ¥DnY4•5/ _wandbruntime‚14689  _runtime‚14689¬ˆÿ(–5€P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2630277002543104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2616742408368128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137226100 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18303762483200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚17299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _step‚1424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137226098 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2648580765026304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297497175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2630277002543104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18304103822336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67547269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784287497396059e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18304103822336 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160261098 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18505277661184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137226100 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160271075 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18505195375616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18275843295682 _runtime‚14689.521845179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37244257 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚579125721704448ýÜÆ) (—5…P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18304103822336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67547269 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784287497396059e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18304103822336 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160261098 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18505277661184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137226100 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160271075 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18505195375616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18275843295682 _runtime‚14689.521845179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37244257 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚579125721704448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2630277002543104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2616742408368128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137226100 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18303762483200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚17299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _step‚1424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137226098 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2648580765026304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297497175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2630277002543104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 è‘Ì'4˜5/ _wandbruntime‚14690  _runtime‚14690t·Ïõ'™5‹N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18305116456448 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2630424243572736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚579159131850752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚579159150462976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2635166130978766 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚597665347859456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18304775110144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160270150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842874980078132e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2648729377068544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137233730 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2616888936079360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18276854349404 _step‚1425 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30304483 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67550849 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18277194899406 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2630424260612096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37246623 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37246366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚17299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297513873 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297503880 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2616888919040000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2648729377068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2648729018682880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297513873 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _runtime‚14690.133656705 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160270150 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2630424243572736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137233732 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67551109 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160280141 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18305116456448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297503880 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2630424260612096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37246623 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37246366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚17299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297513873 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297503880 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2616888919040000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2648729377068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2648729018682880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297513873 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _runtime‚14690.133656705 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160270150 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2630424243572736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137233732 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67551109 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160280141 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18305116456448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297503880 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2635166130978766 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚597665347859456 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18304775110144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160270150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842874980078132e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2648729377068544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137233730 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2616888936079360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18276854349404 _step‚1425 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30304483 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67550849 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18277194899406‘ ƒþ¢4›5/ _wandbruntime‚14690  _runtime‚14690*(òÎ œ5‡ %loss/total_loss‚6.904027798326526 0loss/FRt1/assignment_loss‚1.4747433554401714 3loss/FBt1/assignment_loss‚0.0011042464046862538 1loss/SRqqt1/detection_loss‚0.7382389111484498 _runtime‚14690.136275639 0loss/FRt2/assignment_loss‚0.7365831326505992 2loss/SRqqt1/assignment_loss‚0.3416345626079681 1loss/FBt2/assignment_loss‚0.18932521224008664 epoch‚43 $ _timestamp‚1.784287498010895e+09 0loss/FBt2/detection_loss‚0.40027200224839393 2loss/SRqqt2/assignment_loss‚0.3994602699539666 /loss/FRt1/detection_loss‚0.4722643987973348 /loss/FRt2/detection_loss‚1.1676854548635034 1loss/SRqqt2/detection_loss‚0.5557575527201878 /loss/FBt1/detection_loss‚0.4269586992511774 _step‚1426 trainer/global_step‚17299¢ÓA®5Œ %loss/total_loss‚6.904027798326526 0loss/FRt1/assignment_loss‚1.4747433554401714 3loss/FBt1/assignment_loss‚0.0011042464046862538 1loss/SRqqt1/detection_loss‚0.7382389111484498 _runtime‚14690.136275639 0loss/FRt2/assignment_loss‚0.7365831326505992 2loss/SRqqt1/assignment_loss‚0.3416345626079681 1loss/FBt2/assignment_loss‚0.18932521224008664 epoch‚43 $ _timestamp‚1.784287498010895e+09 0loss/FBt2/detection_loss‚0.40027200224839393 2loss/SRqqt2/assignment_loss‚0.3994602699539666 /loss/FRt1/detection_loss‚0.4722643987973348 /loss/FRt2/detection_loss‚1.1676854548635034 1loss/SRqqt2/detection_loss‚0.5557575527201878 /loss/FBt1/detection_loss‚0.4269586992511774 _step‚1426 trainer/global_step‚17299’ uo­ä“ž5‚(:ˆ “¢èÒÊîë¤ gpu.0.gpu‚40.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ5»½Ÿ5‚(: “¢èÒ¶Þ†¦ network.recv‚ 3.896495e+06& disk.nvme0n1.out‚55782.00146484375 disk.sda.out‚ 604.484375 disk.dm-1.in‚ 0.0078125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 604.484375# disk.nvme0n1.in‚144962.07421875 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144962.07421875! disk.rbd1.out‚120.63720703125$ disk./.usageGB‚536.3923721313477( disk.nvme0c0n1.out‚55782.00146484375 disk.rbd3.in‚0* proc.memory.percent‚17.039871215820312 disk.sda.in‚ 0.0078125 disk.dm-1.out‚ 604.484375 disk.rbd3.out‚0$ memory_percent‚97.82829284667969$ proc.memory.availableMB‚355.8125 proc.cpu.threads‚17* disk./.usagePercent‚35.997742156912224 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚144962.07421875 proc.memory.rssMB‚ 2791.8125 disk.sda3.in‚ 0.0078125# disk.dm-0.out‚55782.00146484375 cpu‚26.009652362753805 network.sent‚ 2.7535488e+07 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625k¶ûíž 5‚(:“ ¢¢èÒà¶Ô¢ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.762800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZúTA¡5‚(:‰ ¢¢èÒ‘ÌŒ¥% disk.nvme0c0n1.in‚144962.07421875 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 604.9765625 disk.rbd3.in‚0 disk.dm-0.out‚55798.89453125 disk.sda.out‚ 604.9765625 disk.sda2.in‚0! disk.rbd1.out‚120.63720703125* proc.memory.percent‚17.039871215820312 cpu‚26.009358116745073 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚144962.07421875# disk.nvme0n1.out‚55798.89453125 disk.dm-1.in‚ 0.0078125% proc.memory.availableMB‚ 355.65625 proc.memory.rssMB‚ 2791.8125* disk./.usagePercent‚35.997672778870445$ disk./.usageGB‚536.3913383483887 network.sent‚ 2.7585747e+07 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚55798.89453125# memory_percent‚97.8292465209961# disk.nvme0n1.in‚144962.07421875 disk.sda2.out‚0 disk.sda3.out‚ 604.9765625 disk.rbd1.in‚0.0625 network.recv‚ 3.901107e+06 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0{‹4¢5/ _wandbruntime‚14729  _runtime‚14729ÃÛ2Ì£5Æ ' lr-AdamW/pg1‚2.1016439316609346e-05 ' lr-AdamW/pg2‚2.1016439316609346e-05 trainer/global_step‚17349 % _timestamp‚1.7842875374058964e+09 _runtime‚14729.531511037 _step‚1427Ý{“}Ѥ5Ë trainer/global_step‚17349 % _timestamp‚1.7842875374058964e+09 _runtime‚14729.531511037 _step‚1427 ' lr-AdamW/pg1‚2.1016439316609346e-05 ' lr-AdamW/pg2‚2.1016439316609346e-05“ AÁ?4¥5/ _wandbruntime‚14729  _runtime‚14729ú×}«(¦5€P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160725527 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚599353771849728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30389988 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18354799324160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 _step‚1428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚14729.532766545 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚580797159630848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160725527 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2655993512009728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37347569 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2624067959001088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137607700 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18326800332480 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2637639071064064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2637639054024704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18354457985024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37347308 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2655993512009728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚599353673049088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580797141018624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137607700 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2637639054024704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2637639071064064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18354799324160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67737560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2642394401744318 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18354457985024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18556612218880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚17349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160715550 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137607698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30389991 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2637639071064064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2637639054024704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18354457985024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37347308 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2655993512009728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚599353673049088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580797141018624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137607700 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2637639054024704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2637639071064064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18354799324160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67737560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2642394401744318 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18354457985024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18556612218880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚17349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160715550 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137607698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30389991 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2655993512009728” i2Ãj ¨5‚(:• ±¢èÒÁ¯—¥ disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0n1.in‚144962.07421875 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144962.07421875 disk.sda2.out‚0 disk.rbd1.in‚0.0625 network.recv‚ 3.904626e+06 disk.sda1.out‚0' disk.nvme0c0n1.out‚55815.9287109375 disk.sda.in‚ 0.0078125 disk.rbd3.out‚0 network.sent‚ 2.7590782e+07) disk./.usagePercent‚35.99858749379028 disk.drbd2165.in‚0 disk.rbd0.in‚0! disk.rbd1.out‚120.69970703125 disk.rbd3.in‚0( proc.memory.availableMB‚ 355.41796875% disk.nvme0n1.out‚55815.9287109375 disk.sda2.in‚0 disk.sda3.out‚ 605.20703125$ memory_percent‚97.83070087432861 proc.memory.rssMB‚ 2791.8125 disk.sda.out‚ 605.20703125 disk.sda3.in‚ 0.0078125 disk.dm-0.in‚144962.07421875* proc.memory.percent‚17.039871215820312$ disk./.usageGB‚536.4049682617188 disk.dm-1.in‚ 0.0078125" disk.dm-0.out‚55815.9287109375 disk.dm-1.out‚ 605.20703125 cpu‚26.009077629770303 proc.cpu.threads‚17Y16“©5‚(:ˆ ±¢èÒÊÛ§¹ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚270.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kõ‘¡4ª5/ _wandbruntime‚14730  _runtime‚14730WšÃÂ'«5‹N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67741091 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580830569777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18557635816960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2656141765666304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18355470611968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30391733 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚599388112829952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2637786312093696 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2637786295054336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30391736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚599388224206336 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298339932 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚17349 _runtime‚14730.144798203 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18355811958272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298349925 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18355811958272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137615330 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2642541923469912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2656142124051968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160734593 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18327470836312 % _timestamp‚1.7842875380189822e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67741363 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137615332 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2637786295054336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137615332 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298349925 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160724602 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160724602 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2642542281059274 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2624214469672960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2624214452633600 _step‚1429 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 298339932 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160734593 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67741091 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580830569777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18557635816960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2656141765666304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18355470611968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30391733 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚599388112829952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2637786312093696 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2637786295054336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30391736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚599388224206336 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298339932 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚17349 _runtime‚14730.144798203 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18355811958272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298349925 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18355811958272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137615330 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2642541923469912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2656142124051968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160734593 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18327470836312 % _timestamp‚1.7842875380189822e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67741363 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137615332 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2637786295054336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137615332 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298349925 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160724602 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160724602 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2642542281059274 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2624214469672960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2624214452633600 _step‚1429 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 298339932 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0• Q%ºº4­5/ _wandbruntime‚14730  _runtime‚147300Ûoí®5‹ 0loss/FRt2/assignment_loss‚0.7864881999096007 4loss/FBt2/assignment_loss‚0.00013229350482587185 /loss/FRt1/detection_loss‚0.5150364442957953 epoch‚43 0loss/FRt1/assignment_loss‚1.5101977214807831 3loss/SRqqt1/assignment_loss‚0.24104887161116825 /loss/FRt2/detection_loss‚1.2153194434203376 /loss/FBt2/detection_loss‚0.5782922805083217 $ _timestamp‚1.784287538022011e+09 _step‚1430 2loss/SRqqt2/assignment_loss‚0.4576929606226198 2loss/FBt1/assignment_loss‚0.011815445017858897 1loss/SRqqt1/detection_loss‚0.7408192175771944 0loss/FBt1/detection_loss‚0.34418782427923544 1loss/SRqqt2/detection_loss‚0.5224109829966254 &loss/total_loss‚6.9234416852243665 trainer/global_step‚17349 _runtime‚14730.147723571Í02C¯5 1loss/SRqqt2/detection_loss‚0.5224109829966254 &loss/total_loss‚6.9234416852243665 trainer/global_step‚17349 _runtime‚14730.147723571 0loss/FRt2/assignment_loss‚0.7864881999096007 4loss/FBt2/assignment_loss‚0.00013229350482587185 /loss/FRt1/detection_loss‚0.5150364442957953 epoch‚43 0loss/FRt1/assignment_loss‚1.5101977214807831 3loss/SRqqt1/assignment_loss‚0.24104887161116825 /loss/FRt2/detection_loss‚1.2153194434203376 /loss/FBt2/detection_loss‚0.5782922805083217 $ _timestamp‚1.784287538022011e+09 _step‚1430 2loss/SRqqt2/assignment_loss‚0.4576929606226198 2loss/FBt1/assignment_loss‚0.011815445017858897 1loss/SRqqt1/detection_loss‚0.7408192175771944 0loss/FBt1/detection_loss‚0.34418782427923544– *¯“°5‚(:ˆ À¢èÒ‡…å¢ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚235.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.0894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆÑdL ±5‚(:• À¢èÒÞ¬¤ network.sent‚ 2.7643135e+07# disk.dm-0.out‚55833.72021484375 disk.sda3.out‚ 605.38671875 disk.dm-1.in‚ 0.0078125' proc.memory.availableMB‚ 355.4140625 proc.cpu.threads‚17 disk.sda.out‚ 605.38671875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0# disk.nvme0n1.in‚144962.07421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.039871215820312 cpu‚26.00878410995371 disk.sda.in‚ 0.0078125$ memory_percent‚97.83072471618652 proc.memory.rssMB‚ 2791.8125$ disk./.usageGB‚536.4116325378418 disk.rbd0.out‚0% disk.nvme0c0n1.in‚144962.07421875 disk.sda2.in‚0 disk.dm-1.out‚ 605.38671875& disk.nvme0n1.out‚55833.72021484375( disk./.usagePercent‚35.9990347389526( disk.nvme0c0n1.out‚55833.72021484375 disk.sda3.in‚ 0.0078125! disk.rbd1.out‚120.69970703125 disk.rbd3.out‚0 disk.dm-0.in‚144962.07421875 network.recv‚ 3.909042e+06 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0¤`U“²5‚(:ˆ Ï¢èÒÊÂÓ¢ gpu.0.gpu‚79.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 @镳5‚(:ý Ï¢èÒÑÒñ¤ disk.sda3.in‚ 0.0078125 proc.cpu.threads‚17 network.recv‚ 3.913283e+06 disk.dm-0.out‚55865.35546875 disk.sda2.in‚0% proc.memory.availableMB‚ 355.40625 proc.memory.rssMB‚ 2791.8125 cpu‚26.00848618108435# disk.nvme0n1.out‚55865.35546875# disk.nvme0n1.in‚144962.07421875% disk.nvme0c0n1.in‚144962.07421875% disk.nvme0c0n1.out‚55865.35546875 disk.rbd1.in‚0.0625$ memory_percent‚97.83077239990234 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.sda.out‚607.375 disk.sda3.out‚607.375* disk./.usagePercent‚36.000054160953944 disk.dm-1.out‚607.375! disk.rbd1.out‚120.76220703125 disk.dm-0.in‚144962.07421875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125 network.sent‚ 2.7695511e+07$ disk./.usageGB‚536.4268226623535 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.039871215820312 disk.rbd3.out‚0-e8w4´5/ _wandbruntime‚14769  _runtime‚147695êˆÊµ5Ä & lr-AdamW/pg1‚2.024255263720668e-05 & lr-AdamW/pg2‚2.024255263720668e-05 trainer/global_step‚17399 % _timestamp‚1.7842875772852256e+09 _runtime‚14769.410768568 _step‚1431!϶5É trainer/global_step‚17399 % _timestamp‚1.7842875772852256e+09 _runtime‚14769.410768568 _step‚1431 & lr-AdamW/pg1‚2.024255263720668e-05 & lr-AdamW/pg2‚2.024255263720668e-05— Ùw ò4·5/ _wandbruntime‚14769  _runtime‚14769àÌC› ¸5P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137989298 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2645001105506304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137989298 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18607872774144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2645001122545664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137989300 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2663406258993152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚Â`Él0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30477238 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2631393492594688 _runtime‚14769.411973041 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137989298 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18607872774144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2645001122545664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137989300 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2663406258993152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30477238 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2631393492594688 _runtime‚14769.411973041 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67932318 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18608800648704 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2631539986227200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30478986 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18608889218560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18378427873222 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2645148363575296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299175984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30478983 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137996930 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137996930 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2645148346535936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299185977 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582502026315776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚14770.024978942 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601110808352256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601110915534336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299175984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67932054 trainer/global_step‚17399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2631540003266560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137996932 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161189045 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0n¥§ñ ¼5N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18608889218560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18378427873222 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2645148363575296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299175984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30478983 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137996930 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137996930 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2645148346535936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299185977 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582502026315776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚14770.024978942 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601110808352256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601110915534336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299175984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67932054 trainer/global_step‚17399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2631540003266560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137996932 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161189045 4.DeviceStatsMonitor.on_train_batch_end/numÆVµö_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2649918431139782 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚582502007703552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18406507460096 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2649918073550420 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137996932 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67932318 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18608800648704 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2631539986227200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30478986 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0™ ®rƒ—4½5/ _wandbruntime‚14770  _runtime‚14770o‹ï² ¾5… %loss/total_loss‚6.513250454431124 2loss/SRqqt2/assignment_loss‚0.3994012728300568 /loss/FRt2/detection_loss‚1.1515947586060085 1loss/SRqqt2/detection_loss‚0.6239845091967852 trainer/global_step‚17399 % _timestamp‚1.7842875779018142e+09 _runtime‚14770.027479452 /loss/FBt1/detection_loss‚0.2576661580516939 epoch‚43 /loss/FRt1/assignment_loss‚1.431788977873218 0loss/FRt2/assignment_loss‚0.7233021038278336 2loss/SRqqt1/assignment_loss‚0.3317208532879352 1loss/FBt1/assignment_loss‚0.00079942176447787 /loss/FBt2/detection_loss‚0.3759351013043973 _step‚1434 3loss/FBt2/assignment_loss‚0.0023963169980098322 /loss/FRt1/detection_loss‚0.5102639041905039 0loss/SRqqt1/detection_loss‚0.704397076500204x¶r!¿5Š %loss/total_loss‚6.513250454431124 2loss/SRqqt2/assignment_loss‚0.3994012728300568 /loss/FRt2/detection_loss‚1.1515947586060085 1loss/SRqqt2/detection_loss‚0.6239845091967852 trainer/global_step‚17399 % _timestamp‚1.7842875779018142e+09 _runtime‚14770.027479452 /loss/FBt1/detection_loss‚0.2576661580516939 epoch‚43 /loss/FRt1/assignment_loss‚1.431788977873218 0loss/FRt2/assignment_loss‚0.7233021038278336 2loss/SRqqt1/assignment_loss‚0.3317208532879352 1loss/FBt1/assignment_loss‚0.00079942176447787 /loss/FBt2/detection_loss‚0.3759351013043973 _step‚1434 3loss/FBt2/assignment_loss‚0.0023963169980098322 /loss/FRt1/detection_loss‚0.5102639041905039 0loss/SRqqt1/detection_loss‚0.704397076500204š È+‰“À5‚(:ˆ Þ¢èÒÍ¿¶£ gpu.0.gpu‚98.0 gpu.0.memory‚35+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;^;I Á5‚(:þ Þ¢èÒÅ·¾¥ network.sent‚ 2.7700594e+07 disk.sda3.in‚ 0.0078125 disk.sda3.out‚610 disk.dm-1.in‚ 0.0078125$ disk./.usageGB‚536.4285583496094 disk.dm-0.in‚144962.07421875# disk.dm-0.out‚55892.00146484375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚55892.00146484375! disk.rbd1.out‚120.76220703125& disk.nvme0n1.out‚55892.00146484375 disk.sda2.in‚0 disk.sda2.out‚0* disk./.usagePercent‚36.000170644382024 disk.sda.in‚ 0.0078125 disk.dm-1.out‚610 proc.cpu.threads‚17# disk.nvme0n1.in‚144962.07421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.out‚610$ memory_percent‚97.83084392547607( proc.memory.availableMB‚ 355.39453125 proc.memory.rssMB‚ 2791.8125* proc.memory.percent‚17.039871215820312 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚144962.07421875 network.recv‚ 3.916514e+06 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 cpu‚26.008222359221698È’”Â5‚(:’ í¢èÒ¯ý³¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.087' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.81739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëå°ÖÃ5‚(:“ í¢èÒ¡ò¤ disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚55912.41650390625 disk.rbd3.in‚0 disk.dm-0.in‚144962.07421875* proc.memory.percent‚17.039871215820312 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚120.82470703125 disk.rbd3.out‚0# disk.dm-0.out‚55912.41650390625) disk./.usagePercent‚36.00060201707721 disk.sda3.out‚ 613.02734375 disk.dm-1.in‚ 0.0078125 cpu‚26.007896056831104% disk.nvme0c0n1.in‚144962.07421875 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 network.recv‚ 3.920937e+06# disk./.usageGB‚536.434986114502 disk.sda.out‚ 613.02734375# memory_percent‚97.8311538696289 proc.cpu.threads‚17# disk.nvme0n1.in‚144962.07421875& disk.nvme0n1.out‚55912.41650390625 disk.sda1.in‚0 disk.dm-1.out‚ 613.02734375 network.sent‚ 2.7763628e+07 disk.sda2.out‚0% proc.memory.availableMB‚ 355.34375 proc.memory.rssMB‚ 2791.8125q)+˜Ä5‚(:’ ü¢èÒ®¹Á¢ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸeE$Å5‚(:“ ü¢èÒÝëÚ¤# disk./.usageGB‚536.437629699707% disk.nvme0c0n1.in‚144962.07421875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 614.5234375 network.sent‚ 2.776783e+07( disk.nvme0c0n1.out‚55934.66650390625 disk.sda.in‚ 0.0078125 disk.sda.out‚ 614.5234375 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚120.82470703125 disk.rbd3.out‚0# disk.nvme0n1.in‚144962.07421875( proc.memory.availableMB‚ 355.53515625* proc.memory.percent‚17.039871215820312 network.recv‚ 3.923414e+06 disk.rbd3.in‚0# disk.dm-0.out‚55934.66650390625 disk.sda1.out‚0 proc.memory.rssMB‚ 2791.8125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.in‚144962.07421875$ memory_percent‚97.82998561859131* disk./.usagePercent‚36.000779430298444 disk.rbd0.in‚0 disk.sda3.out‚ 614.5234375& disk.nvme0n1.out‚55934.66650390625 cpu‚26.00761939539678 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625à4Æ5/ _wandbruntime‚14809  _runtime‚14809béœ/ÊÇ5Ä trainer/global_step‚17449 % _timestamp‚1.7842876172515862e+09 _runtime‚14809.377132829 _step‚1435 & lr-AdamW/pg1‚1.948254511453839e-05 & lr-AdamW/pg2‚1.948254511453839e-05í,–ÏÈ5É & lr-AdamW/pg2‚1.948254511453839e-05 trainer/global_step‚17449 % _timestamp‚1.7842876172515862e+09 _runtime‚14809.377132829 _step‚1435 & lr-AdamW/pg1‚1.948254511453839e-05› ÿ³fô4É5/ _wandbruntime‚14809  _runtime‚14809nŒå÷(Ê5P WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138370898 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2670819364355072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18456190327808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18659001137664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161624454 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7842876172526772e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18455848988672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30564491 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2638719026188288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37554841 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚584140035483648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚584140016871424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299995352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚14809.378454254 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138370900 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68119060 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2657147059494584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602798923140608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18428033306296 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2670819005976576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138370900 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161634431 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18427692756406 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚1436 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2652363174027264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2657146701905334 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2670819005976576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2652363156987904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602799036621312 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2638719009148928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161634431 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18455848988672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18658906269184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18456190327808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300005331 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2652363174027264 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138370898 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2670819364355072 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161634431 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18455848988672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18658906269184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18456190327808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300005331 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2652363174027264 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138370898 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2670819364355072 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18455848988672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30564491 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2638719026188288 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37554841 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚584140035483648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚584140016871424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299995352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚14809.378454254 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138370900 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68119060 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2657147059494584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602798923140608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18428033306296 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2670819005976576 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138370900 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161634431 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18427692756406 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚1436 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2652363174027264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2657146701905334 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2670819005976576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269œ ¢‚A 4Ì5/ _wandbruntime‚14809  _runtime‚14809k)²ãÍ5‹N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18456861615616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18457202961920 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18429044360130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30566236 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18428703810128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18456861615616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2638865536860160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37556885 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842876178654802e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300012036 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2652510415056896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300022029 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚584173464242176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138378532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138378530 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161643497 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚602833370603520 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161633506 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68123121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30566233 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2652510398017536 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138378532 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68122843 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚14809.991331912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2657294223630928 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2638865519820800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚17449 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18660017737728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18659924973568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138378530 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2652510398017536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚584173445629952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602833481979904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591Áq‚c'Î5N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300012036 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2652510415056896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300022029 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚584173464242176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138378532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138378530 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161643497 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚602833370603520 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161633506 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68123121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30566233 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2652510398017536 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138378532 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68122843 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚14809.991331912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2657294223630928 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2638865519820800 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚17449 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18660017737728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18659924973568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138378530 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2652510398017536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚584173445629952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602833481979904 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18456861615616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18457202961920 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18429044360130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30566236 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18428703810128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18456861615616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2638865536860160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37556885 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842876178654802e+09 Vs%4Ï5/ _wandbruntime‚14809  _runtime‚14809-îÚÀÐ5‰ 0loss/FRt1/assignment_loss‚1.5326669553105983 2loss/FBt1/assignment_loss‚0.005471844570361542 0loss/FRt2/assignment_loss‚0.7716223713520175 /loss/FRt1/detection_loss‚0.4860301877704228 /loss/FRt2/detection_loss‚1.3269617905051727 1loss/SRqqt2/detection_loss‚0.6407682116587491 0loss/FBt2/detection_loss‚0.48587663033646783 epoch‚43 trainer/global_step‚17449 _runtime‚14809.993671959 3loss/FBt2/assignment_loss‚0.0007172819100618057 _step‚1438 2loss/SRqqt1/assignment_loss‚0.3247017314298096 2loss/SRqqt2/assignment_loss‚0.4672537682876935 1loss/SRqqt1/detection_loss‚0.7384613597636766 /loss/FBt1/detection_loss‚0.2720120050217331 %loss/total_loss‚7.052544137916764 % _timestamp‚1.7842876178683326e+09ÿ0i¡Ñ5Ž % _timestamp‚1.7842876178683326e+09 0loss/FRt1/assignment_loss‚1.5326669553105983 2loss/FBt1/assignment_loss‚0.005471844570361542 0loss/FRt2/assignment_loss‚0.7716223713520175 /loss/FRt1/detection_loss‚0.4860301877704228 /loss/FRt2/detection_loss‚1.3269617905051727 1loss/SRqqt2/detection_loss‚0.6407682116587491 0loss/FBt2/detection_loss‚0.48587663033646783 epoch‚43 trainer/global_step‚17449 _runtime‚14809.993671959 3loss/FBt2/assignment_loss‚0.0007172819100618057 _step‚1438 2loss/SRqqt1/assignment_loss‚0.3247017314298096 2loss/SRqqt2/assignment_loss‚0.4672537682876935 1loss/SRqqt1/detection_loss‚0.7384613597636766 /loss/FBt1/detection_loss‚0.2720120050217331 %loss/total_loss‚7.052544137916764ž EŠ¶]›Ò5‚(: ‹£èÒ¸ª¤£ gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚280.405' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.08099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:kdÂ#Ó5‚(:˜ ‹£èÒÕŸ€¥ disk.rbd3.out‚0 disk.sda3.out‚ 616.24609375! disk.rbd1.out‚120.88720703125& disk.nvme0n1.out‚55942.34326171875 proc.memory.rssMB‚ 2791.8125 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 616.24609375 disk.sda2.in‚0 disk.sda2.out‚0* disk./.usagePercent‚36.001223603370356% disk.nvme0c0n1.in‚144962.07421875( disk.nvme0c0n1.out‚55942.34326171875 disk.dm-1.in‚ 0.0078125( proc.memory.availableMB‚ 355.26171875 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125 disk.dm-1.out‚ 616.24609375* proc.memory.percent‚17.039871215820312 cpu‚26.007324741350054# disk.dm-0.out‚55942.34326171875 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.7773022e+07 network.recv‚ 3.92703e+06 disk.rbd3.in‚0$ disk./.usageGB‚536.4442481994629 disk.dm-0.in‚144962.07421875# disk.nvme0n1.in‚144962.07421875 disk.drbd2165.out‚0 disk.rbd0.in‚0$ memory_percent‚97.83165454864502Îœ8“Ô5‚(:ˆ š£èÒͶɢ gpu.0.gpu‚52.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÂçLÕ5‚(:‘ š£èÒ”Œì¤ disk.sda2.out‚0( proc.memory.availableMB‚ 355.50390625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.0078125 proc.cpu.threads‚17 network.sent‚ 2.7836097e+07 disk.rbd1.in‚0.0625$ memory_percent‚97.83017635345459 proc.memory.rssMB‚ 2791.8125 disk.dm-1.in‚ 0.0078125# disk.nvme0n1.in‚144962.07421875& disk.nvme0c0n1.out‚55956.267578125 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 616.55078125 disk.sda2.in‚0) disk./.usagePercent‚36.00155641316486$ disk./.usageGB‚536.4492073059082 disk.dm-1.out‚ 616.55078125 disk.rbd3.in‚0$ disk.nvme0n1.out‚55956.267578125 disk.drbd2165.in‚0! disk.rbd1.out‚120.88720703125* proc.memory.percent‚17.039871215820312 disk.dm-0.in‚144962.07421875 disk.drbd2165.out‚0 cpu‚26.00704912454496 network.recv‚ 3.931634e+06 disk.rbd3.out‚0! disk.dm-0.out‚55956.267578125 disk.sda1.in‚0% disk.nvme0c0n1.in‚144962.07421875 disk.sda.out‚ 616.55078125ç·6%4Ö5/ _wandbruntime‚14849  _runtime‚14849ÍârÒÌ×5Æ % _timestamp‚1.7842876573355315e+09 _runtime‚14849.461117271 _step‚1439 ' lr-AdamW/pg1‚1.8736465562467506e-05 ' lr-AdamW/pg2‚1.8736465562467506e-05 trainer/global_step‚17499'ÄÑØ5Ë ' lr-AdamW/pg1‚1.8736465562467506e-05 ' lr-AdamW/pg2‚1.8736465562467506e-05 trainer/global_step‚17499 % _timestamp‚1.7842876573355315e+09 _runtime‚14849.461117271 _step‚1439Ÿ ä_Ù4Ù5/ _wandbruntime‚14849  _runtime‚14849æ š‘"Ú5ÿO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2646044542742528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2678232111338496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18478309243314 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚604521554112512 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2659725225508864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2659725208469504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300841383 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18710198377472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68309755 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 $ _timestamp‚1.784287657336639e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138752500 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18506544490496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2678231752960000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30651741 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18710099314688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2678231752960000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚585811454797824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2678232111338496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138752498 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_tS¾¨train_batch_start/requested_bytes.all.freed‚2664522851985842 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2646044559781888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300831404 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162088883 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37657742 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37658014 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888|ÑÓë (Û5„P YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2678232111338496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18478309243314 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚604521554112512 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2659725225508864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2659725208469504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300841383 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18710198377472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68309755 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 $ _timestamp‚1.784287657336639e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138752500 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18506544490496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2678231752960000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30651741 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18710099314688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2678231752960000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚585811454797824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2678232111338496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138752498 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2664522851985842 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2646044559781888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300831404 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162088883 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37657742 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37658014 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2646044542742528  'ÃÞÜ5‚(:’ ©£èÒç´Ã¢ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%aÝTÝ5‚(:’ ©£èÒæ懤% disk.nvme0n1.out‚55969.4599609375' disk.nvme0c0n1.out‚55969.4599609375 disk.drbd2165.out‚0 disk.sda.in‚ 0.0078125* disk./.usagePercent‚36.001794244164174 disk.dm-1.in‚ 0.0078125' proc.memory.availableMB‚ 355.5078125 disk.dm-0.in‚144962.07421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2791.81640625# disk./.usageGB‚536.452751159668# disk.nvme0n1.in‚144962.07421875 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17" disk.dm-0.out‚55969.4599609375 disk.sda1.out‚0% disk.nvme0c0n1.in‚144962.07421875 disk.sda.out‚ 617.734375* proc.memory.percent‚17.039895057678223 network.sent‚ 2.7841186e+07 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625! disk.rbd1.out‚120.94970703125 disk.rbd3.in‚0 disk.drbd2165.in‚0 network.recv‚ 3.935296e+06 disk.dm-1.out‚ 617.734375$ memory_percent‚97.83015251159668 disk.rbd0.out‚0 cpu‚26.006726002413803 disk.sda2.in‚0 disk.sda3.out‚ 617.734375 disk.rbd3.out‚0÷S³4Þ5/ _wandbruntime‚14850  _runtime‚14850£'“¸'ß5ŠN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2646191053414400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162087958 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300858081 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18507898463744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2664670373711436 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30653483 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18507898463744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37659971 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18711137784320 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2659872466538496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162087958 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300848088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚604556122231296 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2664670731300798 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2659872466538496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚585844883556352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68313192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162097949 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18507557117440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138760132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18479660847038 $ _timestamp‚1.784287657947601e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2678380365002240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2646191070453760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18479320297036 trainer/global_step‚17499 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138760130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68313457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚604556021340672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚585844902168576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300848088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138760132 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2678380006616576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300858081 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2659872449499136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138760130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2659872449499136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18711220062720ì±#!S%à5N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300858081 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2659872449499136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138760130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2659872449499136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18711220062720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2646191053414400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162087958 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300858081 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18507898463744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2664670373711436 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30653483 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18507898463744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37659971 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18711137784320 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2659872466538496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162087958 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300848088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚604556122231296 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2664670731300798 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2659872466538496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚585844883556352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68313192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162097949 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18507557117440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138760132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18479660847038 $ _timestamp‚1.784287657947601e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2678380365002240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2646191070453760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18479320297036 trainer/global_step‚17499 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138760130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68313457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚604556021340672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚585844902168576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300848088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138760132 REDeviceStatsMonitor.on_train_batch_end/active_bytQå€ÅÂes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2678380006616576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¡ ÆÍ4á5/ _wandbruntime‚14850  _runtime‚14850bØGâ5 %loss/total_loss‚6.484826569550242 0loss/FRt2/assignment_loss‚0.7557013323097754 4loss/FBt1/assignment_loss‚1.9976021353556523e-05 0loss/SRqqt2/detection_loss‚0.585020169747755 /loss/FBt2/detection_loss‚0.3195897930879788 1loss/SRqqt1/detection_loss‚0.7761173124781362 epoch‚43 _step‚1442 0loss/FRt1/assignment_loss‚1.4549335425526364 4loss/FBt2/assignment_loss‚2.4044145329890726e-05 0loss/FRt1/detection_loss‚0.46268315090865286 0loss/FBt1/detection_loss‚0.29617543709736643 trainer/global_step‚17499 3loss/SRqqt1/assignment_loss‚0.30973938171786003 2loss/SRqqt2/assignment_loss‚0.4237213872165587 /loss/FRt2/detection_loss‚1.1011010422668377 % _timestamp‚1.7842876579502757e+09 _runtime‚14850.075612934 sÚã5’ 0loss/FRt1/assignment_loss‚1.4549335425526364 4loss/FBt2/assignment_loss‚2.4044145329890726e-05 0loss/FRt1/detection_loss‚0.46268315090865286 0loss/FBt1/detection_loss‚0.29617543709736643 trainer/global_step‚17499 3loss/SRqqt1/assignment_loss‚0.30973938171786003 2loss/SRqqt2/assignment_loss‚0.4237213872165587 /loss/FRt2/detection_loss‚1.1011010422668377 % _timestamp‚1.7842876579502757e+09 _runtime‚14850.075612934 %loss/total_loss‚6.484826569550242 0loss/FRt2/assignment_loss‚0.7557013323097754 4loss/FBt1/assignment_loss‚1.9976021353556523e-05 0loss/SRqqt2/detection_loss‚0.585020169747755 /loss/FBt2/detection_loss‚0.3195897930879788 1loss/SRqqt1/detection_loss‚0.7761173124781362 epoch‚43 _step‚1442¢ Ê“£ä5‚(:’ ¸£èÒ£Ž¼£ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÆ)Aå5‚(:Œ ¸£èÒܪ¥ disk.rbd1.in‚0.0625 disk.dm-0.in‚ 144962.140625 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125 disk.sda.out‚ 620.1484375 disk.rbd3.out‚0! disk.dm-0.out‚55995.130859375 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 144962.140625$ disk.nvme0n1.out‚55995.130859375 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 3.939298e+06# proc.memory.rssMB‚ 2791.81640625* proc.memory.percent‚17.039895057678223 cpu‚26.00642956433725 network.sent‚ 2.7893495e+07) disk./.usagePercent‚36.00289917268196! disk.nvme0n1.in‚ 144962.140625 disk.sda2.in‚0 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚120.94970703125$ disk./.usageGB‚536.4692153930664 disk.rbd0.out‚0 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0 disk.dm-1.out‚ 620.1484375 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 620.1484375$ memory_percent‚97.83031940460205( proc.memory.availableMB‚ 355.48046875& disk.nvme0c0n1.out‚55995.130859375PÑbâ“æ5‚(:ˆ Ç£èÒš®Â¢ gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PM6wç5‚(:‘ Ç£èÒÁ¡Ã¤! disk.rbd1.out‚121.01220703125 disk.rbd3.in‚0$ disk./.usageGB‚536.4673805236816( disk.nvme0c0n1.out‚56019.35107421875 disk.dm-1.in‚ 0.0078125 disk.sda1.out‚0# proc.memory.rssMB‚ 2791.81640625* proc.memory.percent‚17.039895057678223 disk.rbd0.out‚0 disk.sda3.out‚ 623.8046875 disk.dm-0.in‚ 144962.140625 cpu‚26.00615804306658 disk.sda.out‚ 623.8046875 disk.rbd3.out‚0& disk.nvme0n1.out‚56019.35107421875 disk.drbd2165.in‚0 network.sent‚ 2.7945882e+07 disk.dm-1.out‚ 623.8046875$ memory_percent‚97.83182144165039& proc.memory.availableMB‚ 355.234375 network.recv‚ 3.943399e+06# disk.nvme0c0n1.in‚ 144962.140625 disk.sda3.in‚ 0.0078125 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚56019.35107421875 proc.cpu.threads‚17! disk.nvme0n1.in‚ 144962.140625 disk.drbd2165.out‚0* disk./.usagePercent‚36.002776033057984 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.rbd1.in‚0.0625’ï*d4è5/ _wandbruntime‚14889  _runtime‚14889#ó¢JÉé5à _runtime‚14889.244487576 _step‚1443 & lr-AdamW/pg1‚1.800436190028944e-05 & lr-AdamW/pg2‚1.800436190028944e-05 trainer/global_step‚17549 $ _timestamp‚1.784287697118916e+09Â]4ÛÎê5È $ _timestamp‚1.784287697118916e+09 _runtime‚14889.244487576 _step‚1443 & lr-AdamW/pg1‚1.800436190028944e-05 & lr-AdamW/pg2‚1.800436190028944e-05 trainer/global_step‚17549£ fýá4ë5/ _wandbruntime‚14889  _runtime‚14889º:y˜(ì5P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚606244205233664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2671899002066350 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2653370076336128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18528925730222 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2653370093375488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139134098 _step‚1444 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18761207417344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68499558 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2667087259951104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162533358 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162543335 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162543335 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2667087276990464 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2685644499943424 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2667087276990464 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2685644858321920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842876971199694e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18557239992320 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2667087259951104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18557239992320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162543335 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2667087276990464 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2685644499943424 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2667087276990464 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2685644858321920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 % _timestamp‚1.7842876971199694e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18557239992320 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2667087259951104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18557239992320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚606244205233664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2671899002066350 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2653370076336128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18528925730222 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2653370093375488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139134098 _step‚1444 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18761207417344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0¤ ;Ì?µ4î5/ _wandbruntime‚14889  _runtime‚14889°`Æï5ŒN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162542410 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587516321482752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30740733 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30740736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18762214308864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2667234500980736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2685793111985664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139141730 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18529936783944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2667234500980736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301694133 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2653516604047360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68503325 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139141730 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚14889.857491804 _step‚1445 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162552401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18762315461632 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2672046881381306 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2667234518020096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68503591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 301694133 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37762592 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 301684140 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139141732 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18530277333946 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚1215N"L 4 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18558593965568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162542410 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2685792753600000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842876977317145e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18558252619264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872’!g'ð5‘N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2667234500980736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2685793111985664 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139141730 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18529936783944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2667234500980736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301694133 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2653516604047360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68503325 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139141730 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚14889.857491804 _step‚1445 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162552401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18762315461632 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2672046881381306 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2667234518020096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68503591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 301694133 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37762592 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 301684140 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139141732 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18530277333946 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18558593965568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162542410 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2685792753600000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7842876977317145e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18558252619264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162542410 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587516321482752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30740733 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30740736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18762214308864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002¥ 'ž¢\4ñ5/ _wandbruntime‚14889  _runtime‚14889^Q”¨ò5‹ 0loss/FRt1/assignment_loss‚1.5153408138326896 3loss/SRqqt2/assignment_loss‚0.46607867756747134 /loss/FRt1/detection_loss‚0.5180834829465722 _runtime‚14889.860217679 /loss/FRt2/detection_loss‚1.2625075458589594 1loss/SRqqt2/detection_loss‚0.5826248752640782 0loss/FBt1/detection_loss‚0.36290220140344515 $ _timestamp‚1.784287697734877e+09 %loss/total_loss‚7.050876319262214 0loss/FRt2/assignment_loss‚0.6008249433664438 2loss/FBt1/assignment_loss‚0.020983505520654734 4loss/FBt2/assignment_loss‚1.0907136284821206e-05 1loss/SRqqt1/detection_loss‚0.9542012708171064 /loss/FBt2/detection_loss‚0.4496436892300153 trainer/global_step‚17549 _step‚1446 3loss/SRqqt1/assignment_loss‚0.31767440631849225 epoch‚43h¯ó5 trainer/global_step‚17549 _step‚1446 3loss/SRqqt1/assignment_loss‚0.31767440631849225 epoch‚43 0loss/FRt1/assignment_loss‚1.5153408138326896 3loss/SRqqt2/assignment_loss‚0.46607867756747134 /loss/FRt1/detection_loss‚0.5180834829465722 _runtime‚14889.860217679 /loss/FRt2/detection_loss‚1.2625075458589594 1loss/SRqqt2/detection_loss‚0.5826248752640782 0loss/FBt1/detection_loss‚0.36290220140344515 $ _timestamp‚1.784287697734877e+09 %loss/total_loss‚7.050876319262214 0loss/FRt2/assignment_loss‚0.6008249433664438 2loss/FBt1/assignment_loss‚0.020983505520654734 4loss/FBt2/assignment_loss‚1.0907136284821206e-05 1loss/SRqqt1/detection_loss‚0.9542012708171064 /loss/FBt2/detection_loss‚0.4496436892300153¦ ëˆàå“ô5‚(:ˆ Ö£èÒ¬àϤ gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚280.694' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302503508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2674449311432704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2674449328472064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2674449311432704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2693057246926848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139515698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589154349262848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30826241 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37862930 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68689168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37863208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18579542217130 _runtime‚14929.314668211 _step‚1448 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302503508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2674449311432704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2674449328472064 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2674449311432704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2693057246926848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139515698 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589154349262848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30826241 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37862930 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68689168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37863208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18579542217130 _runtime‚14929.314668211 _step‚1448 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68692944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139523332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚608001008305152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30827983 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2679422673872452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2679423031461814 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18580553270852 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37865224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37864961 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚17599 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2674596569501696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚589187778021376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18580893820854 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163006853 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18608948121088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162996862 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302520192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2693205500583424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302520192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162996862 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18608948121088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163006853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302530185 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68693210 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚608001117584384 _step‚1449 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842877378061614e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68692944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139523332 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚608001008305152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30827983 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2679422673872452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2679423031461814 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18580553270852 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37865224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37864961 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚17599 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2674596569501696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚589187778021376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18580893820854 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163006853 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18608948121088 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162996862 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302520192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2693205500583424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302520192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162996862 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18608948121088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163006853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302530185 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68693210 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚608001117584384 _step‚1449 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842877378061614e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0© Ñ Ú4ƒ6/ _wandbruntime‚14929  _runtime‚14929ÚÑ鱄6‹ 3loss/FBt1/assignment_loss‚0.0006180089581480022 1loss/SRqqt1/detection_loss‚0.8385602783035249 1loss/SRqqt2/detection_loss‚0.5592745389182735 epoch‚43 _runtime‚14929.933212074 4loss/FBt2/assignment_loss‚0.00013573724330662174 % _timestamp‚1.7842877378078704e+09 _step‚1450 0loss/FRt1/detection_loss‚0.49691815884156243 .loss/FRt2/detection_loss‚1.091342118947846 0loss/FBt1/detection_loss‚0.40179590602752535 $loss/total_loss‚6.59391851365474 trainer/global_step‚17599 /loss/FBt2/detection_loss‚0.3190818588154798 0loss/FRt1/assignment_loss‚1.4481319953064893 0loss/FRt2/assignment_loss‚0.6909285074677961 3loss/SRqqt1/assignment_loss‚0.30611828132625124 2loss/SRqqt2/assignment_loss‚0.4410131234985366ºŽUû…6 4loss/FBt2/assignment_loss‚0.00013573724330662174 % _timestamp‚1.7842877378078704e+09 _step‚1450 0loss/FRt1/detection_loss‚0.49691815884156243 .loss/FRt2/detection_loss‚1.091342118947846 0loss/FBt1/detection_loss‚0.40179590602752535 $loss/total_loss‚6.59391851365474 trainer/global_step‚17599 /loss/FBt2/detection_loss‚0.3190818588154798 0loss/FRt1/assignment_loss‚1.4481319953064893 0loss/FRt2/assignment_loss‚0.6909285074677961 3loss/SRqqt1/assignment_loss‚0.30611828132625124 2loss/SRqqt2/assignment_loss‚0.4410131234985366 3loss/FBt1/assignment_loss‚0.0006180089581480022 1loss/SRqqt1/detection_loss‚0.8385602783035249 1loss/SRqqt2/detection_loss‚0.5592745389182735 epoch‚43 _runtime‚14929.933212074ª Á۠퓆6‚(:ˆ ƒ¤èÒ×Àä¥ gpu.0.gpu‚58.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Ï ‡6‚(:„ ƒ¤èÒäÅ­§ disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125) proc.memory.percent‚17.07000732421875 network.recv‚ 3.958261e+06$ disk./.usageGB‚536.4934005737305! disk.rbd1.out‚121.13720703125! disk.dm-0.out‚56082.935546875 proc.memory.rssMB‚2796.75! disk.nvme0n1.in‚ 144962.140625 disk.sda1.out‚0& disk.nvme0c0n1.out‚56082.935546875 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 629.1953125' proc.memory.availableMB‚ 350.7109375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 629.1953125 proc.cpu.threads‚17$ disk.nvme0n1.out‚56082.935546875 disk.dm-1.in‚ 0.0078125 network.sent‚ 2.8037481e+07) disk./.usagePercent‚36.00452226044904 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 629.1953125 cpu‚26.00754118633076 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 144962.140625$ memory_percent‚97.85943031311035# disk.nvme0c0n1.in‚ 144962.140625‚œ¡¨4ˆ6/ _wandbruntime‚14947  _runtime‚14947€œ/Ɖ6{ 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23115499317646027 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9156632423400879 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6307685375213623 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5426914691925049 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7753034234046936 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2637239694595337 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8724129796028137 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20815184712409973 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20746919512748718 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6276010870933533 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6643713116645813 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2173633873462677 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7681846618652344 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19892007112503052 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2741076946258545 *jet/accuracy_1_of_2‚0.7899687886238098 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0238564ÓUÍ602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_4‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857453107833862 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5272922515869141 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2609274685382843 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6585362553596497 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 +jet/accuracy_3_of_4‚0.07847467809915543 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 *jet/accuracy_2_of_4‚0.7984643578529358 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8994080424308777 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.627768337726593 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2637239694595337 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 +particle/specificity‚0.9035332798957825 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8486124873161316 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6059532761573792 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.37318354845046997 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07054143399000168 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23729605972766876 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02215036377310753 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2397366762161255 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 *jet/accuracy_2_of_2‚0.1589706838130951 _step‚1451 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3298245370388031 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49759092926979065 )loss/val_total_loss‚8.078096951390444 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5838170647621155 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024438364431262016 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6629815697669983 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3752671778202057 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24164345860481262 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942677617073059 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7279383540153503 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042978931218385696 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35821670293807983 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 particle/accuracy_1_of_3‚1 _runtime‚14947.627837006 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5724344849586487 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5986672639846802 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6056480407714844 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5834634900093079 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7951912879943848 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6596465110778809 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21633560955524445 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_3‚0.9244745373725891 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6210373640060425 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6665154695510864 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7278532981872559 +jet/accuracy_3_of_3‚0.05619187653064728 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6394056677818298 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04305606707930565 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7222222089767456 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27571338415145874 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8724129796028137 (particle/accuracy‚0.8410683870315552 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9549952149391174 +particle/sensitivity‚0.6126686334609985 'particle/f_score‚0.6234256625175476 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6295248866081238 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2098766267299652 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7646932005882263 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.646620512008667 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7338057160377502 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416666626930237 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2637118995189667 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5133333206176758 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 trainer/global_step‚17599 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5217103362083435 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚1 epoch‚43 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6351982355117798 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2745373845100403 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6590994596481323 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886239171028137 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25545498728752136 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5986722707748413 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26036810874938965 *jet/accuracy_1_of_4‚0.9830800294876099 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32104992866516113 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995073139667511 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6933324933052063 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7315192222595215 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45313185453414917 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7356197834014893 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2014772742986679 particle/accuracy_3_of_3‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23715157806873322 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035453785210847855 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7222222089767456 $ _timestamp‚1.784287755501635e+09 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7190918922424316 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6215202212333679 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8547785878181458 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6966472268104553 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7621220350265503 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.037100743502378464 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.724503219127655 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024263961240649223 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35854464769363403 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6248435974121094 *jet/accuracy_2_of_3‚0.5722455978393555 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5981816053390503 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5318997502326965 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8014306426048279 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2486058622598648 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022018233314156532 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2507404685020447 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27390289306640625 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6056480407714844 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2474507987499237 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6288613677024841 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6628370881080627 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08207976073026657 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6046087145805359 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29724180698394775 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7382086515426636 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22287389636039734 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33133015036582947 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031Úà˜=Š6’{ 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6966472268104553 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7621220350265503 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.037100743502378464 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.724503219127655 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024263961240649223 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35854464769363403 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6248435974121094 *jet/accuracy_2_of_3‚0.5722455978393555 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5981816053390503 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5318997502326965 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8014306426048279 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2486058622598648 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022018233314156532 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2507404685020447 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27390289306640625 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6056480407714844 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2474507987499237 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6288613677024841 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6628370881080627 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08207976073026657 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6046087145805359 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29724180698394775 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7382086515426636 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22287389636039734 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33133015036582947 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23115499317646027 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9156632423400879 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6307685375213623 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5426914691925049 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7753034234046936 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2637239694595337 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8724129796028137 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20815184712409973 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20746919512748718 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6276010870933533 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6643713116645813 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2173633873462677 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7681846618652344 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19892007112503052 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2741076946258545 *jet/accuracy_1_of_2‚0.7899687886238098 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_4‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857453107833862 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5272922515869141 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2609274685382843 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6585362553596497 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 +jet/accuracy_3_of_4‚0.07847467809915543 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 *jet/accuracy_2_of_4‚0.7984643578529358 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8994080424308777 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.627768337726593 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2637239694595337 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 +particle/specificity‚0.9035332798957825 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8486124873161316 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6059532761573792 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.37318354845046997 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07054143399000168 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23729605972766876 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02215036377310753 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2397366762161255 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 *jet/accuracy_2_of_2‚0.1589706838130951 _step‚1451 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3298245370388031 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49759092926979065 )loss/val_total_loss‚8.078096951390444 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5838170647621155 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024438364431262016 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6629815697669983 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3752671778202057 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24164345860481262 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942677617073059 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7279383540153503 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042978931218385696 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35821670293807983 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 particle/accuracy_1_of_3‚1 _runtime‚14947.627837006 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5724344849586487 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5986672639846802 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6056480407714844 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5834634900093079 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7951912879943848 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6596465110778809 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21633560955524445 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_3‚0.9244745373725891 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6210373640060425 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6665154695510864 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7278532981872559 +jet/accuracy_3_of_3‚0.05619187653064728 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6394056677818298 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04305606707930565 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7222222089767456 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27571338415145874 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8724129796028137 (particle/accuracy‚0.8410683870315552 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9549952149391174 +particle/sensitivity‚0.6126686334609985 'particle/f_score‚0.6234256625175476 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6295248866081238 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2098766267299652 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7646932005882263 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.646620512008667 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7338057160377502 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416666626930237 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2637118995189667 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5133333206176758 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 trainer/global_step‚17599 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5217103362083435 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚1 epoch‚43 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6351982355117798 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2745373845100403 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6590994596481323 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886239171028137 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25545498728752136 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5986722707748413 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26036810874938965 *jet/accuracy_1_of_4‚0.9830800294876099 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32104992866516113 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995073139667511 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6933324933052063 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7315192222595215 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45313185453414917 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7356197834014893 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2014772742986679 particle/accuracy_3_of_3‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23715157806873322 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035453785210847855 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7222222089767456 $ _timestamp‚1.784287755501635e+09 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7190918922424316 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6215202212333679 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8547785878181458 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146« DºÑ€“‹6‚(:ˆ ’¤èÒÖŽ”£ gpu.0.gpu‚86.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SfèŒ6‚(: ’¤èÒÀñ·¥ disk.rbd3.in‚0% disk.nvme0n1.out‚56098.5068359375 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚56098.5068359375 disk.sda3.out‚ 629.86328125) disk./.usagePercent‚36.00494876900108 disk.dm-1.out‚ 629.86328125 disk.rbd3.out‚0" disk.dm-0.out‚56098.5068359375 disk.sda1.in‚0 disk.sda3.in‚ 0.0078125 disk.sda1.out‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2800.359375# disk.nvme0c0n1.in‚ 144962.140625 disk.sda.in‚ 0.0078125# disk./.usageGB‚536.499755859375 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 144962.140625 disk.rbd0.out‚0 disk.sda.out‚ 629.86328125 disk.sda2.in‚0* proc.memory.percent‚17.092037200927734 cpu‚26.009190513597957 network.recv‚ 3.962554e+06 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚121.13720703125! disk.nvme0n1.in‚ 144962.140625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 346.62109375 proc.cpu.threads‚17 network.sent‚ 2.8087518e+07 disk.sda2.out‚0$ memory_percent‚97.88439273834229œìŸ¹‘6‚(:† ¡¤èÒ¤¢ gpu.0.gpu‚39.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚203.92' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¢T²Ž6‚(: ¡¤èÒ¡”’¤# disk.nvme0c0n1.in‚ 144962.140625 disk.sda3.out‚ 631.4765625 disk.dm-0.in‚ 144962.140625$ memory_percent‚97.88236618041992 disk.sda1.out‚0' disk.nvme0c0n1.out‚56110.3798828125 disk.sda.in‚ 0.0078125 disk.dm-1.out‚ 631.4765625 disk.rbd0.out‚0 network.sent‚ 2.8144874e+07$ disk./.usageGB‚536.5049934387207 disk.drbd2165.in‚0! disk.rbd1.out‚121.19970703125* proc.memory.percent‚17.092037200927734 cpu‚26.008902983995792 proc.cpu.threads‚17 disk.sda3.in‚ 0.0078125 disk.rbd3.out‚0 network.recv‚ 3.966804e+06 disk.sda1.in‚0& proc.memory.availableMB‚ 346.953125! proc.memory.rssMB‚ 2800.359375 disk.rbd0.in‚0 disk.sda.out‚ 631.4765625 disk.rbd1.in‚0.0625* disk./.usagePercent‚36.005300267345596! disk.nvme0n1.in‚ 144962.140625 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0" disk.dm-0.out‚56110.3798828125% disk.nvme0n1.out‚56110.3798828125Éj“6‚(:“ °¤èÒ#~´‹Ò¢•£ gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.314' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.062799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YØ£™6‚(:‹ °¤èÒ¸†­¦ disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 632.66015625 disk.dm-1.in‚ 0.0078125 cpu‚26.008603430803426 network.sent‚ 2.814991e+07 disk.sda.in‚ 0.0078125$ disk.nvme0n1.out‚56128.849609375& proc.memory.availableMB‚ 346.953125# disk.nvme0c0n1.in‚ 144962.140625 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk./.usageGB‚536.5154037475586& disk.nvme0c0n1.out‚56128.849609375 disk.sda.out‚ 632.66015625 disk.dm-0.in‚ 144962.140625! disk.nvme0n1.in‚ 144962.140625 disk.drbd2165.out‚0$ memory_percent‚97.88236618041992! proc.memory.rssMB‚ 2800.359375 network.recv‚ 3.970008e+06 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 632.66015625* proc.memory.percent‚17.092037200927734) disk./.usagePercent‚36.00599891190654 disk.rbd0.out‚0! disk.rbd1.out‚121.19970703125! disk.dm-0.out‚56128.849609375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0.0625 disk.rbd0.in‚0„…Ô4‘6/ _wandbruntime‚14988  _runtime‚14988« À;Ì’6Æ _step‚1452 ' lr-AdamW/pg1‚1.6582269431547028e-05 ' lr-AdamW/pg2‚1.6582269431547028e-05 trainer/global_step‚17649 % _timestamp‚1.7842877963237002e+09 _runtime‚14988.450507485LжLÑ“6Ë trainer/global_step‚17649 % _timestamp‚1.7842877963237002e+09 _runtime‚14988.450507485 _step‚1452 ' lr-AdamW/pg1‚1.6582269431547028e-05 ' lr-AdamW/pg2‚1.6582269431547028e-05¬ Ù´óQ4”6/ _wandbruntime‚14988  _runtime‚14988c%ôþ(•6P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2683740929564672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18676705540608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚609723370209280 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2688577685166675 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18881689532416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1453 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140017966 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18676364201472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68914321 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2669929823776768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚17649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303530548 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2702417652144640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163512584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2702417293766144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18676364201472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140017966 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30916011 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚609723471107072 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68914053 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2688578042755925 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2669929806737408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18676705540608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2702417652144640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590841699289088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18881771817984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303540527 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140017964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37998045 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18648218979157 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 % _timestamp‚1.7842877963244684e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163512584 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2702417293766144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18676364201472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140017966 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30916011 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚609723471107072 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68914053 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2688578042755925 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2669929806737408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18676705540608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2702417652144640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590841699289088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18881771817984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303540527 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140017964 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37998045 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18648218979157 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 % _timestamp‚1.7842877963244684e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0­ %<4—6/ _wandbruntime‚14989  _runtime‚14989ÐnÕ:'˜6ŠN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 trainer/global_step‚17649 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2683888170594304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303547232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303557225 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68918128 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2683888187633664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140025596 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18882706770432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2702565905808384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18649230032991 $ _timestamp‚1.784287796938108e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68917867 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30917756 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2688725206892269 _runtime‚14989.063956076 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2688725564481631 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163531627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140025596 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2683888187633664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚609757816205824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 303557225 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163521636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18882795340288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163521636 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2702565905808384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2702565547422720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2670076334448640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18677718174720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664¾¹ã¶™6N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚6097579233 žæ_&87904 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163521636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18882795340288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163521636 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2702565905808384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2702565547422720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2670076334448640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18677718174720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 trainer/global_step‚17649 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2683888170594304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303547232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303557225 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68918128 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2683888187633664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140025596 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18882706770432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2702565905808384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18649230032991 $ _timestamp‚1.784287796938108e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68917867 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30917756 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2688725206892269 _runtime‚14989.063956076 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2688725564481631 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163531627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140025596 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2683888187633664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚609757816205824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 303557225 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18698494916175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140399564 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2691102998085632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31003261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚611446139857920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18698835466065 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304376579 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163967036 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2695954192836433 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18727401042432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163977013 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚15030.328021394 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18727059703296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18727059703296 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140399566 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163977013 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18933107734528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69106538 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2709830399128064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38103277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2709830040749568 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚592513137215488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140399566 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38102997 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31003258 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚611446244950016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18933021254656 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842878382022476e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0œ±Mþ¨6†P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69106538 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2709830399128064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38103277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2709830040749568 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚592513137215488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140399566 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38102997 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31003258 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚611446244950016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18933021254656 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842878382022476e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18727401042432 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140399564 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2709830040749568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2677255357370368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2691102981046272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18698494916175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140399564 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2691102998085632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31003261 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚611446139857920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18698835466065 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304376579 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163967036 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2695954192836433 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18727401042432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163977013 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _runtime‚15030.328021394 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18727059703296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18727059703296 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140399566 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163977013 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18933107734528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0± ¤Gî94©6/ _wandbruntime‚15030  _runtime‚15030ý$Ç'ª6‹N D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18728072330240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38105113 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2677401851002880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚17699 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140407196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69110116 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2709978652791808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163986079 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18934132829696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163976088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚611480698803712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2709978294406144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140407196 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18728413676544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7842878388289194e+09 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31005006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2696101356972777 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚15030.954789265 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2691250239115264 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140407198 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18699505969897 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304393277 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2691250239115264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304383284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2691250222075904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18934048454144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2691250222075904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38105366 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304383284 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163976088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304393277 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140407198 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163986079 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18728413676544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69110372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚592546547361792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31005003 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚611480595815936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2677401868042240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088º´Ã1'«6N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2691250239115264 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140407198 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18699505969897 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304393277 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2691250239115264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304383284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2691250222075904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18934048454144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2691250222075904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38105366 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304383284 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163976088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304393277 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140407198 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163986079 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18728413676544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69110372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚592546547361792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31005003 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚611480595815936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2677401868042240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18728072330240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38105113 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2677401851002880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚17699 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140407196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69110116 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2709978652791808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163986079 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18934132829696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163976088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚611480698803712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2709978294406144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140407196 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18728413676544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7842878388289194e+09 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31005006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2696101356972777 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚15030.954789265² ùvÉm4¬6/ _wandbruntime‚15030  _runtime‚15030ÆŽò¨­6Š epoch‚44 3loss/SRqqt2/assignment_loss‚0.46209467371183893 /loss/FRt2/detection_loss‚1.1394395850362902 1loss/SRqqt1/detection_loss‚0.8990434849833884 /loss/FBt2/detection_loss‚0.5899793112527251 %loss/total_loss‚7.116030101423938 % _timestamp‚1.7842878388304856e+09 0loss/FRt2/assignment_loss‚0.7807726159928337 3loss/SRqqt1/assignment_loss‚0.34709439447650525 3loss/FBt1/assignment_loss‚0.0031655817510541813 0loss/FRt1/assignment_loss‚1.3483300673422114 1loss/SRqqt2/detection_loss‚0.6324194025488551 /loss/FBt1/detection_loss‚0.4011276946700915 trainer/global_step‚17699 _runtime‚15030.955814009 2loss/FBt2/assignment_loss‚0.012565701545672133 /loss/FRt1/detection_loss‚0.4999975881124724 _step‚1459†‡ç®6 /loss/FBt1/detection_loss‚0.4011276946700915 trainer/global_step‚17699 _runtime‚15030.955814009 2loss/FBt2/assignment_loss‚0.012565701545672133 /loss/FRt1/detection_loss‚0.4999975881124724 _step‚1459 epoch‚44 3loss/SRqqt2/assignment_loss‚0.46209467371183893 /loss/FRt2/detection_loss‚1.1394395850362902 1loss/SRqqt1/detection_loss‚0.8990434849833884 /loss/FBt2/detection_loss‚0.5899793112527251 %loss/total_loss‚7.116030101423938 % _timestamp‚1.7842878388304856e+09 0loss/FRt2/assignment_loss‚0.7807726159928337 3loss/SRqqt1/assignment_loss‚0.34709439447650525 3loss/FBt1/assignment_loss‚0.0031655817510541813 0loss/FRt1/assignment_loss‚1.3483300673422114 1loss/SRqqt2/detection_loss‚0.6324194025488551³ ûû¸e“¯6‚(:ˆ ì¤èÒ”é÷¤ gpu.0.gpu‚45.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`çÐ¥°6‚(:’ ì¤èÒŒÔǧ disk.rbd3.in‚0 network.sent‚ 2.8228336e+07 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125 disk.rbd1.in‚0.0625! disk.rbd1.out‚121.32470703125! proc.memory.rssMB‚ 2800.359375 cpu‚26.007454877325078 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 144962.140625 disk.sda2.out‚0 disk.dm-1.out‚ 638.31640625 disk.rbd3.out‚0# memory_percent‚97.8858470916748' proc.memory.availableMB‚ 346.3828125 proc.cpu.threads‚17$ disk./.usageGB‚536.5388603210449 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 144962.140625 disk.sda3.in‚ 0.0078125* proc.memory.percent‚17.092037200927734 network.recv‚ 3.984907e+06) disk./.usagePercent‚36.00757310223458! disk.nvme0n1.in‚ 144962.140625& disk.nvme0n1.out‚56188.45654296875 disk.sda2.in‚0 disk.sda3.out‚ 638.31640625 disk.dm-1.in‚ 0.0078125 disk.rbd0.out‚0# disk.dm-0.out‚56188.45654296875( disk.nvme0c0n1.out‚56188.45654296875 disk.sda.out‚ 638.31640625Ô«¨“±6‚(:ˆ û¤èÒÊ廉 gpu.0.gpu‚82.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚262.877' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04H²6‚(:’ û¤èÒ„Œ£¤# disk.dm-0.out‚56197.94873046875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 144962.140625 disk.sda2.in‚0! proc.memory.rssMB‚ 2800.359375) disk./.usagePercent‚36.00793458487293 disk.dm-0.in‚ 144962.140625 disk.rbd0.out‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 638.69921875 network.recv‚ 3.989473e+06 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 638.69921875$ memory_percent‚97.88587093353271( proc.memory.availableMB‚ 346.37890625* proc.memory.percent‚17.092037200927734 cpu‚26.007161335591583 disk.rbd3.in‚0! disk.nvme0n1.in‚ 144962.140625& disk.nvme0n1.out‚56197.94873046875 disk.sda.in‚ 0.0078125 network.sent‚ 2.829145e+07 disk.rbd1.in‚0.0625! disk.rbd1.out‚121.38720703125 disk.sda3.out‚ 638.69921875 disk.dm-1.in‚ 0.0078125 proc.cpu.threads‚17# disk./.usageGB‚536.544246673584 disk.sda1.out‚0( disk.nvme0c0n1.out‚56197.94873046875 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0¸¡Ž^4³6/ _wandbruntime‚15073  _runtime‚15073§ KýÌ´6Æ trainer/global_step‚17749 % _timestamp‚1.7842878811406462e+09 _runtime‚15073.266302645 _step‚1460 ' lr-AdamW/pg1‚1.5216633055812227e-05 ' lr-AdamW/pg2‚1.5216633055812227e-05…­Ž¸Ñµ6Ë % _timestamp‚1.7842878811406462e+09 _runtime‚15073.266302645 _step‚1460 ' lr-AdamW/pg1‚1.5216633055812227e-05 ' lr-AdamW/pg2‚1.5216633055812227e-05 trainer/global_step‚17749´ å© 4¶6/ _wandbruntime‚15073  _runtime‚15073ƦÈt- ·6P _step‚1461 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2703329985327691 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38207906 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2698465049567232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164431465 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613168984052736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚17749 % _timestamp‚1.7842878811418426e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305202652 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18984324528128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2703330342916941 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2684580890963968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140781166 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2717243146111488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164421488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚613168881057792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31090508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38208162 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2717242787732992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 J (¸6†P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2717243146111488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164421488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚613168881057792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31090508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38208162 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2717242787732992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38207906 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2698465049567232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164431465 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613168984052736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚17749 % _timestamp‚1.7842878811418426e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305202652 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18984324528128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2703330342916941 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2684580890963968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140781166µ ÃÀö4¹6/ _wandbruntime‚15073  _runtime‚15073w(ùa'º6‹N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7842878817868574e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305219336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2703477864642647 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140788798 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31092256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305229329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18985435464192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚15073.912814583 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18778767832064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613203338473984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18985353185792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140788796 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164430540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31092253 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2703477507053285 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594217985288192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2717391041389568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613203439364608 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2698612273557504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2698612290596864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164440531 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38210262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2684727384596480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18750122456805 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69302237 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2717391399775232 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305219336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚594218003900416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚17749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164440531 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2698612273557504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18778767832064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305229329 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2698612290596864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18750463006807 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18779109178368 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2717391399775232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69302518 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1462 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚99914±áÁ²»6N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18778767832064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613203338473984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18985353185792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140788796 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164430540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31092253 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2703477507053285 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚594217985288192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2717391041389568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613203439364608 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2698612273557504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2698612290596864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164440531 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38210262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2684727384596480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18750122456805 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69302237 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2717391399775232 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305219336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚594218003900416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚17749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164440531 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2698612273557504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18778767832064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305229329 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2698612290596864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18750463006807 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18779109178368 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2717391399775232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69302518 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1462 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7842878817868574e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305219336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2703477864642647 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140788798 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31092256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305229329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18985435464192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚15073.912814583 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169¶ _òç¢4¼6/ _wandbruntime‚15073  _runtime‚15073½;2­½6 %loss/total_loss‚6.828986853282089 0loss/FRt1/assignment_loss‚1.4821458365623483 3loss/SRqqt1/assignment_loss‚0.33908803189355197 3loss/SRqqt2/assignment_loss‚0.40421311910354507 2loss/FBt1/assignment_loss‚0.001539163842261928 0loss/FRt1/detection_loss‚0.49110053477528864 _step‚1463 epoch‚44 _runtime‚15073.914457975 4loss/FBt2/assignment_loss‚0.00012700647087949722 1loss/SRqqt2/detection_loss‚0.5662501836951495 0loss/FBt1/detection_loss‚0.40881878790224435 trainer/global_step‚17749 % _timestamp‚1.7842878817889936e+09 0loss/FRt2/assignment_loss‚0.7479332764874692 /loss/FRt2/detection_loss‚1.0937777598016185 1loss/SRqqt1/detection_loss‚0.9263907956668971 /loss/FBt2/detection_loss‚0.3676023570808342ÀÑƾ6’ 0loss/FRt1/assignment_loss‚1.4821458365623483 3loss/SRqqt1/assignment_loss‚0.33908803189355197 3loss/SRqqt2/assignment_loss‚0.40421311910354507 2loss/FBt1/assignment_loss‚0.001539163842261928 0loss/FRt1/detection_loss‚0.49110053477528864 _step‚1463 epoch‚44 _runtime‚15073.914457975 4loss/FBt2/assignment_loss‚0.00012700647087949722 1loss/SRqqt2/detection_loss‚0.5662501836951495 0loss/FBt1/detection_loss‚0.40881878790224435 trainer/global_step‚17749 % _timestamp‚1.7842878817889936e+09 0loss/FRt2/assignment_loss‚0.7479332764874692 /loss/FRt2/detection_loss‚1.0937777598016185 1loss/SRqqt1/detection_loss‚0.9263907956668971 /loss/FBt2/detection_loss‚0.3676023570808342 %loss/total_loss‚6.828986853282089· ÕÌWS‘¿6‚(:† Š¥èÒ²ôÌ£ gpu.0.gpu‚92.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L˜MÄÀ6‚(:Œ Š¥èҢɥ disk.dm-0.in‚ 144962.140625# disk.dm-0.out‚56209.99462890625 network.recv‚ 3.992723e+06# disk./.usageGB‚536.549747467041# disk.nvme0c0n1.in‚ 144962.140625 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0! disk.rbd1.out‚121.38720703125( disk./.usagePercent‚36.0083037477373 disk.drbd2165.out‚0 disk.dm-1.out‚ 638.9296875& disk.nvme0n1.out‚56209.99462890625 disk.sda1.out‚0 disk.dm-1.in‚ 0.0078125 cpu‚26.00685200898534 disk.rbd3.out‚0 network.sent‚ 2.8296488e+07 disk.sda.out‚ 638.9296875 disk.sda3.out‚ 638.9296875! disk.nvme0n1.in‚ 144962.140625 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.sda2.in‚0! proc.memory.rssMB‚ 2800.359375* proc.memory.percent‚17.092037200927734( disk.nvme0c0n1.out‚56209.99462890625 disk.sda2.out‚0$ memory_percent‚97.88713455200195& proc.memory.availableMB‚ 346.171875 proc.cpu.threads‚17²RÔØžÁ6‚(:“ ™¥èÒ÷Ṳ̈ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚258.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.762800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.蹇Â6‚(: ™¥èÒªèÌ¥! disk.nvme0n1.in‚ 144962.140625( disk.nvme0c0n1.out‚56220.64306640625 cpu‚26.00656431388827 disk.sda1.out‚0 disk.sda.out‚ 639.265625( proc.memory.availableMB‚ 345.91796875! proc.memory.rssMB‚ 2800.359375* proc.memory.percent‚17.092037200927734 disk.sda2.in‚0 network.recv‚ 3.997157e+06 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚36.00872129602565 disk.sda3.out‚ 639.265625! disk.rbd1.out‚121.44970703125# disk.nvme0c0n1.in‚ 144962.140625 proc.cpu.threads‚17 network.sent‚ 2.8359677e+07 disk.sda1.in‚0# disk.dm-0.out‚56220.64306640625& disk.nvme0n1.out‚56220.64306640625 disk.rbd0.in‚0$ memory_percent‚97.88868427276611 disk.sda.in‚ 0.0078125 disk.dm-1.out‚ 639.265625 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0$ disk./.usageGB‚536.5559692382812 disk.dm-1.in‚ 0.0078125 disk.rbd3.out‚0 disk.dm-0.in‚ 144962.140625–“H.“Ã6‚(:ˆ ¨¥èÒ‘šË¤ gpu.0.gpu‚80.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ÒñÄ6‚(:ƒ ¨¥èÒ̸ž¦! disk.rbd1.out‚121.44970703125$ disk.nvme0c0n1.out‚ 56235.9609375 disk.dm-1.out‚ 640.4453125( proc.memory.availableMB‚ 345.91796875* proc.memory.percent‚17.092037200927734$ disk./.usageGB‚536.5600662231445 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 144962.140625 disk.dm-0.out‚ 56235.9609375 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0078125! proc.memory.rssMB‚ 2800.359375 network.sent‚ 2.836474e+07) disk./.usagePercent‚36.00899624811743 disk.rbd3.out‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 144962.140625 disk.sda2.out‚0 network.recv‚ 4.000383e+06 disk.drbd2165.in‚0 disk.sda.out‚ 640.4453125 disk.sda3.out‚ 640.4453125$ memory_percent‚97.88868427276611 disk.dm-1.in‚ 0.0078125 disk.sda1.out‚0 disk.rbd1.in‚0.0625" disk.nvme0n1.out‚ 56235.9609375 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 cpu‚26.00627501590297! disk.nvme0n1.in‚ 144962.140625îéί4Å6/ _wandbruntime‚15114  _runtime‚15114]áèËÆ6Å _runtime‚15114.192524905 _step‚1464 ' lr-AdamW/pg1‚1.4555096110457285e-05 ' lr-AdamW/pg2‚1.4555096110457285e-05 trainer/global_step‚17799 $ _timestamp‚1.784287922066856e+091\ñ|ÐÇ6Ê _step‚1464 ' lr-AdamW/pg1‚1.4555096110457285e-05 ' lr-AdamW/pg2‚1.4555096110457285e-05 trainer/global_step‚17799 $ _timestamp‚1.784287922066856e+09 _runtime‚15114.192524905¸ n[«4È6/ _wandbruntime‚15114  _runtime‚15114 °^(É6€P YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31177761 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19035667128832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69490191 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2705827084009472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19035589037568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18828792046080 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31177758 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141162766 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2705827101048832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141162764 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2691906407518208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38312684 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141162766 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2691906424557568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69490445 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18799727889991 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2724655534716416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18828792046080 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18800068439881 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚17799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614891680197120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38312433 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164875940 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2705827101048832 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2724655893094912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164885917 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚614891583493632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1465 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18828792046080 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31177758 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141162766 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2705827101048832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141162764 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2691906407518208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38312684 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141162766 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2691906424557568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69490445 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18799727889991 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2724655534716416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18828792046080 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18800068439881 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚17799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614891680197120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38312433 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164875940 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2705827101048832 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2724655893094912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164885917 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚614891583493632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1465 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141170396 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2724804146758656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2705974325039104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306055388 _step‚1466 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2705974342078464 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18829804680192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allS"ocated‚595889441826816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306055388 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164884992 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2724804146758656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18829463333888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69494051 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7842879226801848e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2724803788372992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚595889423214592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19036613996544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31179503 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18829804680192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2710854014723155 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164884992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38314813 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2705974342078464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306065381 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31179506 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164894983 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2705974325039104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154ÐÒ™H'Í6N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141170398 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2705974342078464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306065381 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31179506 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164894983 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2705974325039104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141170396 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2724804146758656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2705974325039104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306055388 _step‚1466 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2705974342078464 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18829804680192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚595889441826816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306055388 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164884992 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2724804146758656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18829463333888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69494051 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7842879226801848e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2724803788372992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚595889423214592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19036613996544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31179503 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18829804680192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2710854014723155 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164884992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38314813 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597527450994688 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2699231958151168 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2699231941111808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18850684926789 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18850344376899 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19086825193984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2713189135491072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18879146208768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165340369 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165330392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7842879617982872e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2732068281699840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1469 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18879146208768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165340369 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31265008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2713189152530432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2718082285488707 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141544366 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597527432382464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69681730 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141544364 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18879487547904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69681993 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141544366 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2732068640078336 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2718082643077957 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2732068640078336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YKDeviceStatsMonitor.on_train_batch_star§t%n t/reserved_bytes.small_pool.allocated‚ 564133888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚15153.927379243 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141544364 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2713189135491072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165330392 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2732068281699840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616614276188672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38416982 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31265011 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2732068281699840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚616614276188672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38416982 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31265011 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597527450994688 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2699231958151168 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2699231941111808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18850684926789 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18850344376899 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19086825193984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2713189135491072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18879146208768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165340369 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165330392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7842879617982872e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2732068281699840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1469 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18879146208768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165340369 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31265008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2713189152530432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2718082285488707 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141544366 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597527432382464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69681730 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141544364 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18879487547904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69681993 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141544366 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2732068640078336 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2718082643077957 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2732068640078336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0½ Ì=µe4Ý6/ _wandbruntime‚15154  _runtime‚15154ýŸ×ª'Þ6‹N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69685815 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚616648730198016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 % _timestamp‚1.7842879624102032e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2732216893742080 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2713336393560064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31266756 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141551996 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18880500182016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31266753 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2718229807214301 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18851355430621 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18880500182016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2713336376520704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306901433 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2732216893742080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165339444 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19087753486336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18880158835712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141551996 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165349435 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚616648614627328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165349435 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2713336393560064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2732216535356416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306901433 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2718230164803663 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306891440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69685544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2699378468823040 _runtime‚15154.536006519 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2699378451783680 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚17849 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2713336376520704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1470 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306891440 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18880500182016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2713336376520704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306901433 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2732216893742080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165339444 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19087753486336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18880158835712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141551996 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165349435 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚616648614627328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165349435 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2713336393560064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2732216535356416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306901433 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2718230164803663 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306891440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69685544 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2699378468823040 _runtime‚15154.536006519 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2699378451783680 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚17849 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2713336376520704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚1470 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306891440 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18929841710592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307720787 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2706557474705408 $ _timestamp‚1.784288001789243e+09 _step‚1473 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚618336833693184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2739481387061760 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165784844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69874113 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2725458435569215 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2720551186972672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38521852 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19137963384320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165784844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38521599 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69873857 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2725458793158465 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18901301413697 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141925964 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599198888921088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165794821 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18900960863807 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2739481028683264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18930183049728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18929841710592 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2706557491744768 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2720551204012032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141925966 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141925964gâÉ (ì6…P _runtime‚15193.915412481 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31352258 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307720787 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2706557474705408 $ _timestamp‚1.784288001789243e+09 _step‚1473 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚618336833693184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2739481387061760 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165784844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69874113 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2725458435569215 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2720551186972672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38521852 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19137963384320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165784844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38521599 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69873857 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2725458793158465 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18901301413697 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141925964 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599198888921088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165794821 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18900960863807 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2739481028683264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18930183049728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18929841710592 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2706557491744768 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2720551204012032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141925966 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141925964 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599198870308864 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141925966 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2739481028683264 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307710808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18929841710592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1474 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141933596 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19139074467840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚17899 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2720698445041664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19138979606528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69877696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚618371392147456 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307737485 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38523693 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165803887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165803887 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18930854337536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚15194.525840011 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2739629640725504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18901971917529 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2706704002416640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599232317679616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141933596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31354003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2720698428002304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69877956 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508ªm¥ñ'ï6ŽN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165803887 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18930854337536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _runtime‚15194.525840011 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2739629640725504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18901971917529 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2706704002416640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599232317679616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141933596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31354003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2720698428002304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69877956 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18931195683840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2725605957294809 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 # _timestamp‚1.78428800240002e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141933598 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2720698445041664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1474 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141933596 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19139074467840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚17899 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2720698445041664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19138979606528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69877696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚618371392147456 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307737485 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38523693 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165803887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ¶vo´4ð6/ _wandbruntime‚15194  _runtime‚15194ŸëÍê ñ6‡ epoch‚44 _runtime‚15194.528499471 _step‚1475 3loss/FBt1/assignment_loss‚0.0015537069374928563 0loss/SRqqt2/detection_loss‚0.585754570840158 % _timestamp‚1.7842880024031613e+09 /loss/FRt2/assignment_loss‚0.741221792551119 0loss/FBt1/detection_loss‚0.31220143981144105 0loss/FRt1/assignment_loss‚1.4379219393795335 2loss/SRqqt2/assignment_loss‚0.4324667471308967 1loss/FBt2/assignment_loss‚0.10776514623134456 /loss/FRt2/detection_loss‚1.1984098000270051 /loss/FBt2/detection_loss‚0.7813963899992544 trainer/global_step‚17899 2loss/SRqqt1/assignment_loss‚0.2473523108369918 /loss/FRt1/detection_loss‚0.5499991714172157 1loss/SRqqt1/detection_loss‚0.8208489755572483 &loss/total_loss‚7.2168919907197004Ñê?ò6Œ 0loss/FRt1/assignment_loss‚1.4379219393795335 2loss/SRqqt2/assignment_loss‚0.4324667471308967 1loss/FBt2/assignment_loss‚0.10776514623134456 /loss/FRt2/detection_loss‚1.1984098000270051 /loss/FBt2/detection_loss‚0.7813963899992544 trainer/global_step‚17899 2loss/SRqqt1/assignment_loss‚0.2473523108369918 /loss/FRt1/detection_loss‚0.5499991714172157 1loss/SRqqt1/detection_loss‚0.8208489755572483 &loss/total_loss‚7.2168919907197004 epoch‚44 _runtime‚15194.528499471 _step‚1475 3loss/FBt1/assignment_loss‚0.0015537069374928563 0loss/SRqqt2/detection_loss‚0.585754570840158 % _timestamp‚1.7842880024031613e+09 /loss/FRt2/assignment_loss‚0.741221792551119 0loss/FBt1/detection_loss‚0.31220143981144105à 9Ë“ó6‚(:ˆ ‚¦èÒÿœÒ¢ gpu.0.gpu‚50.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.132' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü_ˆóô6‚(:‘ ‚¦èÒ£™°¥ disk.rbd3.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 346.47265625! proc.memory.rssMB‚ 2800.359375 disk.rbd3.in‚0 disk.dm-0.in‚ 144962.140625 disk.sda.out‚ 646.47265625* proc.memory.percent‚17.092037200927734! disk.nvme0n1.in‚ 144962.140625 disk.drbd2165.in‚0$ memory_percent‚97.88529872894287 network.recv‚ 4.021681e+06 disk.dm-1.out‚ 646.47265625& disk.nvme0n1.out‚56311.35400390625 network.sent‚ 2.851043e+07$ disk./.usageGB‚536.5879669189453 disk.sda2.in‚0! disk.rbd1.out‚121.69970703125# disk.nvme0c0n1.in‚ 144962.140625 cpu‚26.0046006447125 disk.sda2.out‚0 disk.sda3.out‚ 646.47265625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.drbd2165.out‚0) disk./.usagePercent‚36.01086868722286# disk.dm-0.out‚56311.35400390625 disk.sda1.in‚0( disk.nvme0c0n1.out‚56311.35400390625 disk.sda.in‚ 0.0078125 proc.cpu.threads‚17$”´\’õ6‚(:‡ ‘¦èÒÏ°·¢ gpu.0.gpu‚50.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚282.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.499 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö‡ýö6‚(:‘ ‘¦èÒôòð¤ disk.rbd1.in‚0.0625 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 network.sent‚ 2.8573483e+07$ memory_percent‚97.88494110107422% proc.memory.availableMB‚ 346.53125 cpu‚26.004319265225583) disk./.usagePercent‚36.01130927618929$ disk./.usageGB‚536.5945320129395 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125# disk.dm-0.out‚56322.53173828125 disk.sda1.out‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2800.359375 disk.sda.out‚ 646.76953125 disk.dm-1.out‚ 646.76953125! disk.nvme0n1.in‚ 144962.140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 144962.140625 disk.sda2.in‚0 disk.sda3.out‚ 646.76953125 disk.rbd3.in‚0 disk.dm-0.in‚ 144962.140625 disk.sda1.in‚0& disk.nvme0n1.out‚56322.53173828125 disk.rbd3.out‚0* proc.memory.percent‚17.092037200927734 network.recv‚ 4.026336e+06! disk.rbd1.out‚121.76220703125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚56322.53173828125 proc.cpu.threads‚17 <¡“ž÷6‚(:“  ¦èҥľ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐP†ø6‚(:  ¦èÒœŠ¤# disk.nvme0c0n1.in‚ 144962.140625 disk.sda2.out‚0# disk.dm-0.out‚56335.14794921875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 144962.140625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚56335.14794921875 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.rbd3.in‚0& disk.nvme0n1.out‚56335.14794921875 disk.sda1.out‚0 disk.sda3.in‚ 0.0078125* proc.memory.percent‚17.092037200927734 network.sent‚ 2.8578618e+07 disk.sda1.in‚0 disk.sda.out‚ 646.859375 disk.sda3.out‚ 646.859375 disk.dm-1.out‚ 646.859375 cpu‚26.004023453723814 network.recv‚ 4.029917e+06$ disk./.usageGB‚536.5927810668945 disk.drbd2165.out‚0 disk.sda.in‚ 0.0078125' proc.memory.availableMB‚ 346.5234375) disk./.usagePercent‚36.01119176873107! disk.nvme0n1.in‚ 144962.140625 disk.rbd0.out‚0$ memory_percent‚97.88498878479004 disk.rbd0.in‚0! disk.rbd1.out‚121.76611328125 disk.rbd3.out‚0! proc.memory.rssMB‚ 2800.359375m•Ïw4ù6/ _wandbruntime‚15233  _runtime‚15233 0"Ëú6Å ' lr-AdamW/pg2‚1.2656117363012162e-05 trainer/global_step‚17949 $ _timestamp‚1.784288041768661e+09 _runtime‚15233.895412428 _step‚1476 ' lr-AdamW/pg1‚1.2656117363012162e-05ƒ¥¹Ðû6Ê ' lr-AdamW/pg1‚1.2656117363012162e-05 ' lr-AdamW/pg2‚1.2656117363012162e-05 trainer/global_step‚17949 $ _timestamp‚1.784288041768661e+09 _runtime‚15233.895412428 _step‚1476Ä 0¤è#4ü6/ _wandbruntime‚15233  _runtime‚15233ÛͲ%(ý6P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620059439538176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166239296 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2732834585649723 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18980537212416 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2727913238454272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142307566 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70065566 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38626324 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2732834943238973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚15233.895497338 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2746893775666688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18951917900605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1477 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166239296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38626058 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19189131302912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚17949 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2727913255493632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2746893775666688 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18980878551552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308556839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166249273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚600870326847488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚600870308235264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2713883008299008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438˜{§Î¥ þ6†P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620059439538176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166239296 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2732834585649723 TFDeviceStatsMonitor.on_train_batch_start/»Mgreserved_bytes.small_pool.peak‚ 564133888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18980537212416 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2727913238454272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142307566 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70065566 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38626324 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2732834943238973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚15233.895497338 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2746893775666688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18951917900605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1477 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166239296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38626058 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19189131302912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚17949 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2727913255493632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2746893775666688 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18980878551552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308556839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166249273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚600870326847488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚600870308235264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2713883008299008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438Å ?¥YQ4ÿ6/ _wandbruntime‚15234  _runtime‚15234äÕ ‹'€7‹N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2747042387708928 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2747042029323264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18981891185664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70069646 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70069382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2732982464964679 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308573537 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31441256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166248348 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2728060479483904 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2714029518970880 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19190154929152 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 _runtime‚15234.506509209 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18981549839360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308573537 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2714029536010240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31441253 trainer/global_step‚17949 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18952588404437 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2747042387708928 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2728060496523264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308563544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚600903736993792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38628390 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842880423806775e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2728060496523264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18952928954439 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18981549839360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142315196 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18981549839360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308573537 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2714029536010240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31441253 trainer/global_step‚17949 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18952588404437 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2747042387708928 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2728060496523264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308563544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚600903736993792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38628390 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7842880423806775e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2728060496523264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18952928954439 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18981549839360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142315196 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70069646 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70069382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2732982464964679 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308573537 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31441256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166248348 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2728060479483904 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2714029518970880 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438Æ ¯ Á4‚7/ _wandbruntime‚15234  _runtime‚15234It­‹ ƒ7‡ _runtime‚15234.508484258 trainer/global_step‚17949 _step‚1479 3loss/FBt2/assignment_loss‚8.255525530055744e-05 /loss/FRt2/detection_loss‚1.2229461335796745 1loss/SRqqt1/detection_loss‚0.8082595024739548 /loss/FBt1/detection_loss‚0.3241890575426617 epoch‚44 0loss/FRt2/assignment_loss‚0.7692193120556327 2loss/SRqqt2/assignment_loss‚0.4200488353702484 0loss/FRt1/detection_loss‚0.49722035876446685 1loss/SRqqt2/detection_loss‚0.5772362575973378 $loss/total_loss‚6.76177025966908 % _timestamp‚1.7842880423831267e+09 0loss/FRt1/assignment_loss‚1.4247219538188665 2loss/SRqqt1/assignment_loss‚0.3992419140309724 2loss/FBt1/assignment_loss‚0.011484981423010208 .loss/FBt2/detection_loss‚0.307119397756954Íä3š„7Œ trainer/global_step‚17949 _step‚1479 3loss/FBt2/assignment_loss‚8.255525530055744e-05 /loss/FRt2/detection_loss‚1.2229461335796745 1loss/SRqqt1/detection_loss‚0.8082595024739548 /loss/FBt1/detection_loss‚0.3241890575426617 epoch‚44 0loss/FRt2/assignment_loss‚0.7692193120556327 2loss/SRqqt2/assignment_loss‚0.4200488353702484 0loss/FRt1/detection_loss‚0.49722035876446685 1loss/SRqqt2/detection_loss‚0.5772362575973378 $loss/total_loss‚6.76177025966908 % _timestamp‚1.7842880423831267e+09 0loss/FRt1/assignment_loss‚1.4247219538188665 2loss/SRqqt1/assignment_loss‚0.3992419140309724 2loss/FBt1/assignment_loss‚0.011484981423010208 .loss/FBt2/detection_loss‚0.307119397756954 _runtime‚15234.508484258Ç EŽd“…7‚(:ˆ ¯¦èÒÕ‡Ó¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.J± †7‚(:‰ ¯¦èÒ°‰ª¥ disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.0078125 cpu‚26.00373683453641 disk.sda3.out‚ 647.15625! disk.rbd1.out‚121.82861328125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 144962.140625* proc.memory.percent‚17.092037200927734 network.recv‚ 4.033333e+06 disk.dm-1.out‚ 647.15625 disk.rbd0.in‚0$ disk./.usageGB‚536.6071586608887 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0# disk.dm-0.out‚56346.95654296875 disk.sda.out‚ 647.15625 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.dm-0.in‚ 144962.140625& disk.nvme0n1.out‚56346.95654296875 disk.sda1.out‚0 disk.rbd0.out‚0$ memory_percent‚97.88365364074707' proc.memory.availableMB‚ 346.7421875 disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚56346.95654296875! proc.memory.rssMB‚ 2800.359375 proc.cpu.threads‚17 network.sent‚ 2.8583692e+07) disk./.usagePercent‚36.01215666112762! disk.nvme0n1.in‚ 144962.140625£Pþ“‡7‚(:ˆ ¾¦èÒƒ¬œ¤ gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶n• ˆ7‚(:þ ¾¦èÒí¯„¦ cpu‚26.003453002499356 network.recv‚ 4.037594e+06) disk./.usagePercent‚36.01237580357693# disk./.usageGB‚536.610424041748! disk.nvme0n1.in‚ 144962.140625 disk.sda1.out‚0 disk.sda3.out‚ 647.34375! disk.rbd1.out‚121.82861328125 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 144962.140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0.0625$ disk.nvme0n1.out‚56361.013671875$ memory_percent‚97.88360595703125 disk.sda.out‚ 647.34375 proc.cpu.threads‚17& disk.nvme0c0n1.out‚56361.013671875 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 2.8646766e+07 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 144962.140625! proc.memory.rssMB‚ 2800.359375 disk.sda.in‚ 0.0078125 disk.sda1.in‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 647.34375! disk.dm-0.out‚56361.013671875" proc.memory.availableMB‚346.75* proc.memory.percent‚17.092037200927734ñg¬/T‰7‚(:† ͦèҢ䞢 gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.cx¬_F=orrectedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sëv«Š7‚(:‘ ͦèÒõâ¤) disk./.usagePercent‚36.04662013937951" disk.dm-0.out‚56404.0224609375 proc.cpu.threads‚17 network.sent‚ 2.8651837e+07$ disk./.usageGB‚537.1206893920898( proc.memory.availableMB‚ 346.24609375 cpu‚26.003211440600325 disk.sda.out‚ 648.23046875 disk.dm-0.in‚ 144962.140625! disk.nvme0n1.in‚ 144962.140625$ memory_percent‚97.88668155670166! proc.memory.rssMB‚ 2800.359375% disk.nvme0n1.out‚56404.0224609375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 144962.140625 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.drbd2165.out‚0! disk.rbd1.out‚121.90673828125 disk.sda2.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚56404.0224609375 disk.rbd3.out‚0* proc.memory.percent‚17.092037200927734 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.sda2.out‚0 disk.sda3.out‚ 648.23046875 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 648.23046875 disk.rbd3.in‚0 disk.sda1.out‚0 network.recv‚ 4.041244e+06>óí4‹7/ _wandbruntime‚15273  _runtime‚15273ç4ÌŒ7Æ _runtime‚15273.710393789 _step‚1480 ' lr-AdamW/pg1‚1.2051803996867672e-05 ' lr-AdamW/pg2‚1.2051803996867672e-05 trainer/global_step‚17999 % _timestamp‚1.7842880815846753e+09IUÄ Ñ7Ë ' lr-AdamW/pg1‚1.2051803996867672e-05 ' lr-AdamW/pg2‚1.2051803996867672e-05 trainer/global_step‚17999 % _timestamp‚1.7842880815846753e+09 _runtime‚15273.710393789 _step‚1480È CÔ¹4Ž7/ _wandbruntime‚15273  _runtime‚15273U»(7€P UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166703725 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2721208541892608 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2735275306975232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309392891 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19240455614464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19002193837623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142689164 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309382912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38731027 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2721208558931968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166703725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31526761 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70257514 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19240362843136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142689166 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2740211093319481 _runtime‚15273.711678745 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2754306881028608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19031232714240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2735275289935872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142689164 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19031574053376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2740210735730231 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2754306522650112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1481 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2754306881028608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚17999 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2735275289935872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70257788 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31526758 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166693748 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602541746161664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2754306522650112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166703725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31526761 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70257514 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19240362843136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142689166 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2740211093319481 _runtime‚15273.711678745 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2754306881028608 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19031232714240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2735275289935872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142689164 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19031574053376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2740210735730231 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2754306522650112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚1481 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2754306881028608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚17999 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2735275289935872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70257788 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31526758 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166693748 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602541746161664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2754306522650112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0É ¤Nʧ4‘7/ _wandbruntime‚15274  _runtime‚15274Mñ©€'’7‹N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2754454776306688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70261329 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2721355069603840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309399596 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166712791 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309399596 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166702800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19003545441347 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2735422530965504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2735422530965504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚621816551863808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19003204891345 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚15274.321724263 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31528503 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602575193532416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142696798 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19032245341184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309409589 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2735422548004864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142696798 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166702800 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2721355052564480 trainer/global_step‚17999 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142696796 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842880821959333e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚621816663240192 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2740358257455825 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2754455134692352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19032245341184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142696796 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19032586687488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19241469707776 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142696796 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ;4DeviceStatsMonitor.on_train_b‹—Ha%atch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19032586687488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19241469707776 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70261329 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2721355069603840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309399596 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166712791 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309399596 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166702800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19003545441347 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2735422530965504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2735422530965504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚621816551863808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19003204891345 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚15274.321724263 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31528503 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602575193532416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142696798 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19032245341184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309409589 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2735422548004864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142696798 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166702800 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2721355052564480 trainer/global_step‚17999 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142696796 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842880821959333e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚621816663240192 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2740358257455825 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2754455134692352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541Ê ùíó4”7/ _wandbruntime‚15274  _runtime‚15274¤Å•7‹ _runtime‚15274.325347266 0loss/FRt2/assignment_loss‚0.7130651060077879 2loss/FBt1/assignment_loss‚0.008307935836370408 /loss/FBt1/detection_loss‚0.2999745621481272 %loss/total_loss‚6.480101377198439 _step‚1483 0loss/FRt1/assignment_loss‚1.4617477897794462 /loss/FRt2/detection_loss‚1.1497285490482279 1loss/SRqqt2/detection_loss‚0.5811647410283161 epoch‚44 3loss/SRqqt2/assignment_loss‚0.42336791223492515 4loss/FBt2/assignment_loss‚0.00031013953477110334 0loss/FBt2/detection_loss‚0.30542730515253425 trainer/global_step‚17999 2loss/SRqqt1/assignment_loss‚0.2852440192225538 /loss/FRt1/detection_loss‚0.4788594666263443 1loss/SRqqt1/detection_loss‚0.7729038505790353 % _timestamp‚1.7842880821999938e+09d»ò€–7 epoch‚44 3loss/SRqqt2/assignment_loss‚0.42336791223492515 4loss/FBt2/assignment_loss‚0.00031013953477110334 0loss/FBt2/detection_loss‚0.30542730515253425 trainer/global_step‚17999 2loss/SRqqt1/assignment_loss‚0.2852440192225538 /loss/FRt1/detection_loss‚0.4788594666263443 1loss/SRqqt1/detection_loss‚0.7729038505790353 % _timestamp‚1.7842880821999938e+09 _runtime‚15274.325347266 0loss/FRt2/assignment_loss‚0.7130651060077879 2loss/FBt1/assignment_loss‚0.008307935836370408 /loss/FBt1/detection_loss‚0.2999745621481272 %loss/total_loss‚6.480101377198439 _step‚1483 0loss/FRt1/assignment_loss‚1.4617477897794462 /loss/FRt2/detection_loss‚1.1497285490482279 1loss/SRqqt2/detection_loss‚0.5811647410283161Ë tŠ7œž—7‚(:“ ܦèÒíÚ…£ gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]E˜7‚(: ܦèÒó¶û¤ disk.dm-1.out‚ 648.6171875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 144962.140625 disk.sda3.in‚ 0.0078125 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 network.sent‚ 2.8656856e+07) disk./.usagePercent‚36.17108204226344$ disk./.usageGB‚538.9752616882324! disk.rbd1.out‚121.90673828125 disk.rbd0.in‚0* proc.memory.percent‚17.142271995544434 disk.rbd3.in‚0 disk.dm-0.in‚ 144962.140625 disk.sda.out‚ 648.6171875$ memory_percent‚97.93679714202881 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 338.03515625 disk.dm-1.in‚ 0.0078125! disk.dm-0.out‚57313.775390625 disk.drbd2165.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2808.58984375 disk.sda2.in‚0 disk.sda3.out‚ 648.6171875 network.recv‚ 4.044273e+06! disk.nvme0n1.in‚ 144962.140625$ disk.nvme0n1.out‚57313.775390625 proc.cpu.threads‚17 disk.rbd3.out‚0& disk.nvme0c0n1.out‚57313.775390625 cpu‚26.005618739209574Ýïðè4™7/ _wandbruntime‚15291  _runtime‚15291…¸Ã$‘=š7‹{ 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1041666716337204 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.532293438911438 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7373771071434021 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 _runtime‚15291.424460268 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_3_of_4‚0.07847467809915543 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48645129799842834 particle/accuracy_2_of_5‚1 trainer/global_step‚17999 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26344990730285645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916567325592 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 jet/accuracy_1_of_5‚1 (loss/val_total_loss‚8.08770633409703 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6580511927604675 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.730709433555603 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23966963589191437 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5099999904632568 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5909058451652527 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6033020615577698 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7314247488975525 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993925929069519 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6271218061447144 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.595817506313324 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07485214620828629 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.915330171585083 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6286196112632751 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ,jet/accuracy_4_of_4‚0.011098379269242287 particle/accuracy_1_of_2‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25537413358688354 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897577166557312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6620001792907715 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +particle/specificity‚0.9037648439407349 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24859078228473663 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.239272341132164 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6448033452033997 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26252880692481995 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3687184154987335 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021092629060149193 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3295682370662689 jet/accuracy_2_of_5‚1 _step‚1484 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2634359896183014 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03538135439157486 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6617547869682312 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3209361433982849 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022244015708565712 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5710596442222595 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35690227150917053 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20911675691604614 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27352556586265564 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27670562267303467 *jet/accuracy_2_of_3‚0.5727412104606628 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7695996165275574 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6228190064430237 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 particle/accuracy_1_of_4‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_2_of_2‚0.15866178274154663 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45094794034957886 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2634359896183014 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3569296598434448 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6016667485237122 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6387488842010498 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8017664551734924 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3307391107082367 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7954880595207214 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23209264874458313 % _timestamp‚1.7842880992981567e+09 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8451610803604126 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8543776273727417 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20241385698318481 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5478382110595703 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7414965629577637 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6033020615577698 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6948055624961853 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946258068084717 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7213160991668701 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.05679454281926155 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.659899115562439 'particle/accuracy‚0.841390073299408 +particle/sensitivity‚0.6133168935775757 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953467845916748 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7376038432121277 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7497732639312744 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 )jet/accuracy_1_of_2‚0.789733350276947 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.614004373550415 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02239731140434742 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 .particle/accuracy_4_of_4‚0.948646068572998 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22380726039409637 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6605904698371887 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2601267397403717 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8758367896080017 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5970429182052612 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21722914278507233 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.766800045967102 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6265335083007812 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5938225984573364 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20718161761760712 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08726147562265396 *jet/accuracy_1_of_3‚0.9247826933860779 7validation_average_jet_accuracy‚0.37298575043678284 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975671172142029 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6684995889663696 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2737303376197815 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6252253651618958 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8987301588058472 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7218238115310669 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970018267631531 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5815611481666565 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7648417949676514 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21764390170574188 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2967887222766876 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7769463062286377 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )jet/accuracy_2_of_4‚0.806925356388092 'particle/f_score‚0.6241522431373596 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5714307427406311 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25060567259788513 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04603808373212814 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 epoch‚44 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9830800294876099 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6263472437858582 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6315621137619019 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2098766267299652 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.039588600397109985 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.620685875415802 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6915799975395203 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6616604924201965 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24380475282669067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5328617095947266 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27428871393203735 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875767827033997 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23727500438690186 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5226712822914124 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9918553233146667 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04289044812321663 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24887561798095703 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.020968034863471985 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1986975222826004 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8758367896080017 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1hQå³Ô›7{ <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45094794034957886 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2634359896183014 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3569296598434448 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6016667485237122 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6387488842010498 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8017664551734924 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3307391107082367 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7954880595207214 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23209264874458313 % _timestamp‚1.7842880992981567e+09 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8451610803604126 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8543776273727417 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20241385698318481 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5478382110595703 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7414965629577637 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6033020615577698 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6948055624961853 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946258068084717 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7213160991668701 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.05679454281926155 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.659899115562439 'particle/accuracy‚0.841390073299408 +particle/sensitivity‚0.6133168935775757 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953467845916748 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7376038432121277 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7497732639312744 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 )jet/accuracy_1_of_2‚0.789733350276947 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.614004373550415 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02239731140434742 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 .particle/accuracy_4_of_4‚0.948646068572998 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22380726039409637 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6605904698371887 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2601267397403717 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8758367896080017 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5970429182052612 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21722914278507233 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.766800045967102 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6265335083007812 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5938225984573364 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20718161761760712 particle/accuracy_3_of_4‚éõÂ,1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08726147562265396 *jet/accuracy_1_of_3‚0.9247826933860779 7validation_average_jet_accuracy‚0.37298575043678284 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975671172142029 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6684995889663696 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2737303376197815 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6252253651618958 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8987301588058472 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7218238115310669 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970018267631531 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5815611481666565 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7648417949676514 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21764390170574188 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2967887222766876 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7769463062286377 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )jet/accuracy_2_of_4‚0.806925356388092 'particle/f_score‚0.6241522431373596 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5714307427406311 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25060567259788513 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04603808373212814 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 epoch‚44 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9830800294876099 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6263472437858582 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6315621137619019 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2098766267299652 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.039588600397109985 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.620685875415802 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6915799975395203 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6616604924201965 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24380475282669067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5328617095947266 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27428871393203735 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875767827033997 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23727500438690186 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5226712822914124 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9918553233146667 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04289044812321663 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24887561798095703 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.020968034863471985 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1986975222826004 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8758367896080017 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1041666716337204 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.532293438911438 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7373771071434021 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 _runtime‚15291.424460268 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_3_of_4‚0.07847467809915543 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48645129799842834 particle/accuracy_2_of_5‚1 trainer/global_step‚17999 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26344990730285645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916567325592 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 jet/accuracy_1_of_5‚1 (loss/val_total_loss‚8.08770633409703 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6580511927604675 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.730709433555603 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23966963589191437 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5099999904632568 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5909058451652527 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6033020615577698 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7314247488975525 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993925929069519 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6271218061447144 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.595817506313324 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07485214620828629 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.915330171585083 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6286196112632751 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ,jet/accuracy_4_of_4‚0.011098379269242287 particle/accuracy_1_of_2‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25537413358688354 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897577166557312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6620001792907715 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +particle/specificity‚0.9037648439407349 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24859078228473663 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.239272341132164 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6448033452033997 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26252880692481995 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3687184154987335 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021092629060149193 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3295682370662689 jet/accuracy_2_of_5‚1 _step‚1484 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2634359896183014 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03538135439157486 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6617547869682312 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3209361433982849 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022244015708565712 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5710596442222595 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35690227150917053 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20911675691604614 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27352556586265564 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27670562267303467 *jet/accuracy_2_of_3‚0.5727412104606628 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7695996165275574 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6228190064430237 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 particle/accuracy_1_of_4‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +jet/accuracy_2_of_2‚0.15866178274154663Ì ŠPv‘œ7‚(:† ë¦èÒ’¤ñ¢ gpu.0.gpu‚53.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚186.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dfu7‚(:‹ ë¦èÒֵĥ disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.sda.in‚ 0.0078125# proc.memory.rssMB‚ 2808.91796875) disk./.usagePercent‚36.30550161419919 disk.rbd0.out‚0* proc.memory.percent‚17.144274711608887 disk.rbd3.out‚0& disk.nvme0n1.out‚58835.30517578125 disk.drbd2165.out‚0$ memory_percent‚97.93863296508789& proc.memory.availableMB‚ 337.734375$ disk./.usageGB‚540.9782104492188! disk.rbd1.out‚122.03173828125 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0.0625# disk.nvme0c0n1.in‚ 144962.140625( disk.nvme0c0n1.out‚58835.30517578125 disk.sda2.in‚0 disk.sda.out‚ 649.21875 disk.dm-1.out‚ 649.21875 disk.dm-0.in‚ 144962.140625! disk.nvme0n1.in‚ 144962.140625 disk.drbd2165.in‚0 disk.rbd0.in‚0 cpu‚26.007044849700254 network.sent‚ 2.8719924e+07 network.recv‚ 4.048957e+06 disk.sda3.out‚ 649.21875# disk.dm-0.out‚58835.30517578125 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17E´1¨“ž7‚(:ˆ ú¦èÒÓü¬¢ gpu.0.gpu‚67.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ÎY—Ÿ7‚(: ú¦èÒìÁϤ disk.dm-0.in‚ 144962.140625# disk.dm-0.out‚59088.93017578125) disk./.usagePercent‚36.33397886827794 disk.sda3.out‚ 649.6484375 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 cpu‚26.00676453295424 proc.cpu.threads‚17$ disk./.usageGB‚541.4025421142578! disk.rbd1.out‚122.03173828125 disk.drbd2165.out‚0 disk.sda.in‚ 0.0078125 disk.sda.out‚ 649.6484375# proc.memory.rssMB‚ 2808.91796875 network.sent‚ 2.8777332e+07 network.recv‚ 4.052879e+06 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 649.6484375! disk.nvme0n1.in‚ 144962.140625# disk.nvme0c0n1.in‚ 144962.140625 disk.rbd3.in‚0$ memory_percent‚97.94015884399414 disk.sda2.out‚0& disk.nvme0n1.out‚59088.93017578125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚59088.93017578125 disk.drbd2165.in‚0& proc.memory.availableMB‚ 337.484375* proc.memory.percent‚17.1442747116088875³uÁž 7‚(:“ ‰§èÒ¢Ò³¢ gpu.0.gpu‚82.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê Š<¡7‚(: ‰§èÒÔ´í¤ network.sent‚ 2.8782357e+07$ disk./.usageGB‚541.8131561279297 disk.sda3.out‚ 650.16796875 disk.dm-1.out‚ 650.16796875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.dm-0.in‚ 144962.140625' proc.memory.availableMB‚ 337.2421875 disk.sda3.in‚ 0.0078125 disk.rbd3.out‚0 disk.sda.in‚ 0.0078125# proc.memory.rssMB‚ 2808.91796875 cpu‚26.00649695708988 proc.cpu.threads‚17 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 144962.140625 disk.sda.out‚ 650.16796875$ memory_percent‚97.94163703918457* proc.memory.percent‚17.144274711608887) disk./.usagePercent‚36.36153551926357 disk.dm-1.in‚ 0.0078125" disk.dm-0.out‚61638.3798828125 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚61638.3798828125 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚61638.3798828125! disk.rbd1.out‚122.09423828125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 4.05612e+06! disk.nvme0n1.in‚ 144962.140625 disk.sda1.in‚0H «4¢7/ _wandbruntime‚15330  _runtime‚15330ä᭤̣7Æ % _timestamp‚1.7842881386640234e+09 _runtime‚15330.789592213 _step‚1485 ' lr-AdamW/pg1‚1.1461895863936988e-05 ' lr-AdamW/pg2‚1.1461895863936988e-05 trainer/global_step‚18049‚âØѤ7Ë ' lr-AdamW/pg1‚1.1461895863936988e-05 ' lr-AdamW/pg2‚1.1461895863936988e-05 trainer/global_step‚18049 % _timestamp‚1.7842881386640234e+09 _runtime‚15330.789592213 _step‚1485Í "˜'°4¥7/ _wandbruntime‚15330  _runtime‚15330©sã(¦7P UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31616531 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2744566908067840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167218522 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2744566925107200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2730442738700288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842881386650233e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167218522 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19309908199424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70480937 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _step‚1486 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2763666569489408 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2744566908067840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143191432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2749513626339298 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143191430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19070530049760 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167228499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38864670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38864409 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚604229114800128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310409952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70481201 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 % _timestamp‚1.7842881386650233e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167218522 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19309908199424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70480937 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _step‚1486 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2763666569489408 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2744566908067840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143191432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2749513626339298 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143191430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19070530049760 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167228499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38864670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38864409 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚604229114800128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310409952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70481201 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310436629 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70484692 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2730589249372160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70484959 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2749661148065004 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚15331.402568043 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚604262543558656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167237565 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19310856414720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19100674048512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143199064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19071881653484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38866419 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2763815181531648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2744714149097472 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167227574 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310426636 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143199062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167237565 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976«×„ñ'ª7ŽN QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70484692 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2730589249372160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70484959 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2749661148065004 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚15331.402568043 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚604262543558656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167237565 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19310856414720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19100674048512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143199064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19071881653484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38866419 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2763815181531648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2744714149097472 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167227574 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310426636 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143199062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167237565 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2763815181531648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 $ _timestamp‚1.784288139276779e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310426636 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19100674048512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310436629 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Ï ^‹­n4«7/ _wandbruntime‚15331  _runtime‚15331Æ|¬¬7‹ 2loss/FBt1/assignment_loss‚0.006780825971563858 0loss/FRt1/detection_loss‚0.49751974851153946 1loss/SRqqt2/detection_loss‚0.5771925413887022 2loss/SRqqt1/assignment_loss‚0.3481714824725267 1loss/SRqqt1/detection_loss‚0.7168503593271065 /loss/FBt2/detection_loss‚0.3836229359980487 %loss/total_loss‚6.867746566747355 epoch‚45 0loss/FRt2/assignment_loss‚0.7891340608249849 3loss/FBt2/assignment_loss‚7.948163205347382e-06 /loss/FRt2/detection_loss‚1.2283199889127208 0loss/FBt1/detection_loss‚0.40587737626184306 % _timestamp‚1.7842881392799718e+09 _runtime‚15331.405313726 _step‚1488 0loss/FRt1/assignment_loss‚1.4474678385066475 3loss/SRqqt2/assignment_loss‚0.46680146040846554 trainer/global_step‚18049ö‚Bέ7 0loss/FRt2/assignment_loss‚0.7891340608249849 3loss/FBt2/assignment_loss‚7.948163205347382e-06 /loss/FRt2/detection_loss‚1.2283199889127208 0loss/FBt1/detection_loss‚0.40587737626184306 % _timestamp‚1.7842881392799718e+09 _runtime‚15331.405313726 _step‚1488 0loss/FRt1/assignment_loss‚1.4474678385066475 3loss/SRqqt2/assignment_loss‚0.46680146040846554 trainer/global_step‚18049 2loss/FBt1/assignment_loss‚0.006780825971563858 0loss/FRt1/detection_loss‚0.49751974851153946 1loss/SRqqt2/detection_loss‚0.5771925413887022 2loss/SRqqt1/assignment_loss‚0.3481714824725267 1loss/SRqqt1/detection_loss‚0.7168503593271065 /loss/FBt2/detection_loss‚0.3836229359980487 %loss/total_loss‚6.867746566747355 epoch‚45Ð Žc±ý’®7‚(:‡ ˜§èÒ•¯ì¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ߘÙ¯7‚(:‡ ˜§èÒ°Éô¤! disk.nvme0n1.in‚ 144962.140625$ memory_percent‚97.94025421142578! disk.rbd1.out‚122.09423828125 disk.rbd3.in‚0 disk.dm-0.in‚ 144962.140625$ disk.nvme0n1.out‚61718.724609375 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0.0625& disk.nvme0c0n1.out‚61718.724609375 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 650.4609375 disk.dm-1.out‚ 650.4609375# proc.memory.rssMB‚ 2808.91796875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.0078125% proc.memory.availableMB‚ 337.46875 network.sent‚ 2.8787436e+07) disk./.usagePercent‚36.49316000094046 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 144962.140625 network.recv‚ 4.0592e+06$ disk./.usageGB‚543.7744560241699 disk.drbd2165.out‚0 disk.sda.out‚ 650.4609375 disk.sda3.in‚ 0.0078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.144274711608887 cpu‚26.00618179704654! disk.dm-0.out‚61718.724609375¶Y@ °7‚(:’ §§èÒô̺£ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.589999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°wÚ.±7‚(:„ §§èÒ•È¡¤ network.sent‚ 2.8850815e+07# disk.nvme0c0n1.in‚ 144962.140625 disk.sda.in‚ 0.0078125 disk.dm-1.out‚651.25$ disk./.usageGB‚545.5788078308105 disk.drbd2165.out‚0* proc.memory.percent‚17.144274711608887 disk.sda3.out‚651.25 disk.rbd1.in‚0.0625 disk.rbd3.in‚0( proc.memory.availableMB‚ 337.44921875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 144962.140625& disk.nvme0n1.out‚63465.18603515625 disk.sda1.in‚0( disk.nvme0c0n1.out‚63465.18603515625 disk.sda.out‚651.25 disk.dm-1.in‚ 0.0078125 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.rbd1.out‚122.15673828125$ memory_percent‚97.94037342071533# proc.memory.rssMB‚ 2808.91796875 cpu‚26.005918427188615 network.recv‚ 4.063682e+06# disk.dm-0.out‚63465.18212890625 disk.rbd3.out‚0) disk./.usagePercent‚36.61425156463613 disk.dm-0.in‚ 144962.140625 disk.sda3.in‚ 0.0078125ç5†4²7/ _wandbruntime‚15370  _runtime‚15370§Æ©ºË³7Å ' lr-AdamW/pg1‚1.0886430852917294e-05 ' lr-AdamW/pg2‚1.0886430852917294e-05 trainer/global_step‚18099 $ _timestamp‚1.784288178391998e+09 _runtime‚15370.517638748 _step‚1489sE(˜Ð´7Ê ' lr-AdamW/pg1‚1.0886430852917294e-05 ' lr-AdamW/pg2‚1.0886430852917294e-05 trainer/global_step‚18099 $ _timestamp‚1.784288178391998e+09 _runtime‚15370.517638748 _step‚1489Ñ ÝÏ4µ7/ _wandbruntime‚15370  _runtime‚15370w[zöH ¶7ÿO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19150698262528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625261806990336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143573032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167672974 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19121487086558 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70673427 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167682951 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2751928976588800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚15370.522338506 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311246004 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2756889418830556 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167682951 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.Òó÷½freed‚605900534114304 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2737768289333248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31703778 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2751928959549440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19150356923392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 trainer/global_step‚18099 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19150356923392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70673703 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38969922 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19121146536668 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31703781 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605900552726528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143573032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2771079316472832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311255983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2756889776419806 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167672974 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38969649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2771079674851328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19361254263808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19361165686784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438Ü„ÕÝ (·7„P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625261699801088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2751928959549440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143573030 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2771079316472832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311255983 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2756889776419806 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167672974 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38969649 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2771079674851328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19361254263808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19361165686784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19150698262528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625261806990336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143573032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167672974 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19121487086558 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70673427 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167682951 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2751928976588800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚15370.522338506 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311246004 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2756889418830556 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167682951 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605900534114304 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2737768289333248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31703778 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2751928959549440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19150356923392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 trainer/global_step‚18099 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19150356923392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70673703 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38969922 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19121146536668 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31703781 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605900552726528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143573032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70677268 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2737914800005120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311272681 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19362179951104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19151710896640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605933981485056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19122498140392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19151369550336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19362272715264 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167682026 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143580664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625296142823936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143580664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70677527 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38971745 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311262688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311272681 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2757036940556150 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31705526 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7842881791938734e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2752076200579072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625296254200320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2757037298145512 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167692017 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1491 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143580662 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2752076217618432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2752076200579072 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143580662 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚18099 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2752076217618432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚1491 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143580662 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2752076217618432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2752076200579072 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143580662 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚18099 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2752076217618432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70677268 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2737914800005120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311272681 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19362179951104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19151710896640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605933981485056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19122498140392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19151369550336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19362272715264 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167682026 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143580664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625296142823936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143580664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70677527 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38971745 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311262688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311272681 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2757036940556150 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31705526 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7842881791938734e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2752076200579072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625296254200320Ó ¡Ü”C4»7/ _wandbruntime‚15371  _runtime‚15371)—y ¼7‡ 2loss/FBt1/assignment_loss‚0.004038834207312322 3loss/FBt2/assignment_loss‚2.654357006082952e-05 -loss/FBt1/detection_loss‚0.27718556730363 /loss/FBt2/detection_loss‚0.3543061192013456 /loss/FRt1/detection_loss‚0.4387329724723168 %loss/total_loss‚6.974244308238175 0loss/FRt2/assignment_loss‚0.8637250998081305 /loss/FRt2/detection_loss‚1.2659819345908994 epoch‚45 % _timestamp‚1.7842881791965568e+09 _step‚1492 3loss/SRqqt1/assignment_loss‚0.30390958661140843 2loss/SRqqt2/assignment_loss‚0.4378578072080721 1loss/SRqqt1/detection_loss‚0.9765528657571122 1loss/SRqqt2/detection_loss‚0.5317608443317245 trainer/global_step‚18099 _runtime‚15371.321889884 0loss/FRt1/assignment_loss‚1.5201661331761624<õñ½7Œ 0loss/FRt1/assignment_loss‚1.5201661331761624 2loss/FBt1/assignment_loss‚0.004038834207312322 3loss/FBt2/assignment_loss‚2.654357006082952e-05 -loss/FBt1/detection_loss‚0.27718556730363 /loss/FBt2/detection_loss‚0.3543061192013456 /loss/FRt1/detection_loss‚0.4387329724723168 %loss/total_loss‚6.974244308238175 0loss/FRt2/assignment_loss‚0.8637250998081305 /loss/FRt2/detection_loss‚1.2659819345908994 epoch‚45 % _timestamp‚1.7842881791965568e+09 _step‚1492 3loss/SRqqt1/assignment_loss‚0.30390958661140843 2loss/SRqqt2/assignment_loss‚0.4378578072080721 1loss/SRqqt1/detection_loss‚0.9765528657571122 1loss/SRqqt2/detection_loss‚0.5317608443317245 trainer/global_step‚18099 _runtime‚15371.321889884Ô ½M‘¾7‚(:† ¶§èÒÌŸ‡£ gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¨®¿7‚(:” ¶§èÒøÅ̦ disk.sda2.out‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚122.15673828125 disk.drbd2165.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚546.8524169921875 disk.sda3.in‚ 0.0078125 disk.sda.in‚ 0.0078125! disk.nvme0n1.in‚ 144962.140625 disk.sda1.out‚0( disk./.usagePercent‚36.6997245440854 disk.dm-1.in‚ 0.0078125 disk.drbd2165.out‚0 disk.rbd0.in‚0$ memory_percent‚97.94185161590576( proc.memory.availableMB‚ 337.20703125 disk.dm-1.out‚ 651.58203125 disk.rbd3.out‚0& disk.nvme0n1.out‚64563.00244140625 disk.sda1.in‚0( disk.nvme0c0n1.out‚64563.00244140625# proc.memory.rssMB‚ 2808.91796875 cpu‚26.00564507723937 proc.cpu.threads‚17 network.recv‚ 4.066825e+06 disk.sda.out‚ 651.58203125# disk.nvme0c0n1.in‚ 144962.140625 network.sent‚ 2.8868877e+07 disk.sda2.in‚0 disk.sda3.out‚ 651.58203125 disk.rbd3.in‚0 disk.dm-0.in‚ 144962.140625# disk.dm-0.out‚64563.00244140625* proc.memory.percent‚17.144274711608887¸³CGœÀ7‚(:‘ ŧèÒéÿ»£ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì²B!Á7‚(:– ŧèÒÆ—Ö¤ disk.dm-1.in‚ 0.0078125# disk.nvme0c0n1.in‚ 144962.140625 cpu‚26.005370995188226 disk.sda2.in‚0 disk.rbd1.in‚0.0625! disk.nvme0n1.in‚ 144962.140625 disk.sda.in‚ 0.0078125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 651.87890625& disk.nvme0n1.out‚67173.69287109375( proc.memory.availableMB‚ 337.01953125 disk.dm-1.out‚ 651.87890625 disk.drbd2165.in‚0 disk.sda.out‚ 651.87890625 network.sent‚ 2.8918938e+07 network.recv‚ 4.071643e+06$ memory_percent‚97.94299602508545# proc.memory.rssMB‚ 2808.91796875* proc.memory.percent‚17.144274711608887 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚67173.69287109375 disk.rbd3.in‚0! disk.rbd1.out‚122.21923828125# disk.dm-0.out‚67173.69287109375 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0078125 disk.rbd3.out‚0 disk.dm-0.in‚ 144962.140625 disk.rbd0.in‚0) disk./.usagePercent‚36.81706713344964$ disk./.usageGB‚548.6009063720703Û3y†Â7‚(:’ Ô§èÒÌôÛ¢ gpu.0.gpu‚86.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\#ýÃ7‚(:ú Ô§èÒ„‡¸¤$ disk./.usageGB‚550.3436660766602 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 proc.cpu.threads‚17* disk./.usagePercent‚36.934025199497455 disk.sda.out‚652.25# proc.memory.rssMB‚ 2808.91796875 disk.dm-1.out‚652.25 network.sent‚ 2.8923971e+07 disk.dm-0.in‚ 144962.140625 disk.drbd2165.in‚0 disk.rbd0.in‚0# memory_percent‚97.9421854019165! disk.nvme0n1.in‚ 144962.140625# disk.nvme0n1.out‚67189.14453125% disk.nvme0c0n1.out‚67189.14453125* proc.memory.percent‚17.144274711608887 network.recv‚ 4.074978e+06( proc.memory.availableMB‚ 337.15234375 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.out‚652.25 disk.dm-1.in‚ 0.0078125 disk.sda1.out‚0 disk.drbd2165.out‚0 cpu‚26.00510771867084 disk.rbd1.in‚0.0625! disk.rbd1.out‚122.21923828125 disk.dm-0.out‚67189.14453125# disk.nvme0c0n1.in‚ 144962.140625 disk.sda.in‚ 0.0078125‚J±i4Ä7/ _wandbruntime‚15410  _runtime‚15410¡VŽþÌÅ7Æ % _timestamp‚1.7842882183338537e+09 _runtime‚15410.459464868 _step‚1493 ' lr-AdamW/pg1‚1.0325445924850846e-05 ' lr-AdamW/pg2‚1.0325445924850846e-05 trainer/global_step‚18149ÄÚ_¬ÑÆ7Ë ' lr-AdamW/pg1‚1.0325445924850846e-05 ' lr-AdamW/pg2‚1.0325445924850846e-05 trainer/global_step‚18149 % _timestamp‚1.7842882183338537e+09 _runtime‚15410.459464868 _step‚1493Õ vXƒì4Ç7/ _wandbruntime‚15410  _runtime‚15410î–ɹ(È7€P `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19412267273728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143954630 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168127426 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚84120576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2778492063456256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31791031 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2745093805887488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19201052425216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143954630 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2759291011031040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312092035 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143954632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2778492421834752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2759291028070400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚626984323435008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19171763023576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65508352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19172103573466 % _timestamp‚1.7842882183348856e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19201393764352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19412332782080 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1494 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2778492421834752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70864456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2764265568911064 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312082056 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168127426 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143954632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2764265926500314 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2759291028070400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168137403 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2745093822926848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496‰Î> (É7…P XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2759291028070400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168137403 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2745093822926848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19412267273728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143954630 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168127426 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚84120576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2778492063456256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31791031 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2745093805887488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19201052425216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143954630 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2759291011031040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312092035 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143954632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2778492421834752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2759291028070400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚626984323435008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19171763023576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚65508352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19172103573466 % _timestamp‚1.7842882183348856e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19201393764352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19412332782080 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚1494 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2778492421834752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70864456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2764265568911064 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312082056 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168127426 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143954632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2764265926500314Ö þY;·4Ê7/ _wandbruntime‚15411  _runtime‚15411÷ö¤Â‘Ë7ŠN 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168146469 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39075469 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143962262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784288218946715e+09 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2745240333598720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312098740 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70868245 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19202406398464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2778640317112832 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2745240316559360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312108733 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚607605419411456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19172774077298 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31792773 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39075228 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19413274693632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚15411.072518857 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19202065052160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2759438252060672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2759438252060672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚607605400799232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19413365360640 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2764413448226020 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143962262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31792776 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2759438269100032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312098740 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168136478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143962264 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2778640675498496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19202406398464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70868001 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚18149 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168146469 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2778640675498496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312108733 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438báOq'Ì7N D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39075228 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19413274693632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚15411.072518857 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19202065052160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2759438252060672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2759438252060672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚607605400799232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19413365360640 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2764413448226020 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143962262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31792776 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2759438269100032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312098740 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168136478 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143962264 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2778640675498496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19202406398464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70868001 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚18149 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168146469 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2778640675498496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312108733 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168146469 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39075469 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143962262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784288218946715e+09 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2745240333598720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312098740 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70868245 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19202406398464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2778640317112832 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2745240316559360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312108733 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚607605419411456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19172774077298 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31792773 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0× Ä‰¬4Í7/ _wandbruntime‚15411  _runtime‚15411‘É Î7‡ /loss/FBt2/detection_loss‚0.5773094475015468 _runtime‚15411.075119872 _step‚1496 2loss/FBt2/assignment_loss‚0.005315828283757079 /loss/FRt1/detection_loss‚0.5388870024274711 1loss/SRqqt1/detection_loss‚0.8534671705400579 %loss/total_loss‚6.943608962258212 epoch‚45 0loss/FRt2/assignment_loss‚0.7126713417611032 2loss/SRqqt2/assignment_loss‚0.3705730043918543 0loss/FRt1/assignment_loss‚1.4199669546843892 1loss/SRqqt2/detection_loss‚0.6098392961362854 /loss/FBt1/detection_loss‚0.2657888405371108 trainer/global_step‚18149 % _timestamp‚1.7842882189497864e+09 3loss/SRqqt1/assignment_loss‚0.39642283477713586 2loss/FBt1/assignment_loss‚0.013562385662384572 .loss/FRt2/detection_loss‚1.179804855555116PP}DÏ7Œ /loss/FBt2/detection_loss‚0.5773094475015468 _runtime‚15411.075119872 _step‚1496 2loss/FBt2/assignment_loss‚0.005315828283757079 /loss/FRt1/detection_loss‚0.5388870024274711 1loss/SRqqt1/detection_loss‚0.8534671705400579 %loss/total_loss‚6.943608962258212 epoch‚45 0loss/FRt2/assignment_loss‚0.7126713417611032 2loss/SRqqt2/assignment_loss‚0.3705730043918543 0loss/FRt1/assignment_loss‚1.4199669546843892 1loss/SRqqt2/detection_loss‚0.6098392961362854 /loss/FBt1/detection_loss‚0.2657888405371108 trainer/global_step‚18149 % _timestamp‚1.7842882189497864e+09 3loss/SRqqt1/assignment_loss‚0.39642283477713586 2loss/FBt1/assignment_loss‚0.013562385662384572 .loss/FRt2/detection_loss‚1.179804855555116Ø šþžÐ7‚(:“ ã§èÒé¼À¥ gpu.0.gpu‚89.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.354400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìk°Ñ7‚(: ã§èÒùÊå¦ network.sent‚ 2.8929161e+07 disk.sda.in‚ 0.0078125 disk.sda2.in‚0# disk.dm-0.out‚70686.22021484375$ disk.nvme0c0n1.in‚144981.1171875# proc.memory.rssMB‚ 2808.91796875$ disk./.usageGB‚550.4904136657715 disk.rbd3.out‚0& disk.nvme0n1.out‚70686.22021484375 disk.drbd2165.out‚0 disk.sda.out‚ 652.46875 disk.rbd1.in‚0.0625! disk.rbd1.out‚122.28173828125( disk.nvme0c0n1.out‚70686.22021484375 disk.dm-1.out‚ 652.46875" disk.nvme0n1.in‚144981.1171875 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚144981.1171875 disk.rbd0.in‚0( proc.memory.availableMB‚ 337.37890625* proc.memory.percent‚17.144274711608887 cpu‚26.004835848618725 proc.cpu.threads‚17 network.recv‚ 4.07872e+06 disk.sda3.out‚ 652.46875 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0$ memory_percent‚97.94080257415771* disk./.usagePercent‚36.943873553332146 disk.sda3.in‚ 0.0078125 disk.sda1.in‚0·EWžÒ7‚(:“ ò§èÒÓƒ £ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚286.023' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.204600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“B·Ó7‚(: ò§èÒƒ“™¥ disk.rbd0.in‚0# proc.memory.rssMB‚ 2808.91796875 proc.cpu.threads‚17! disk.rbd1.out‚122.28173828125 disk.dm-0.in‚144999.02734375# disk.nvme0n1.out‚70699.53515625 disk.rbd0.out‚0* proc.memory.percent‚17.144274711608887 network.sent‚ 2.8992211e+07 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 652.6328125 disk.drbd2165.out‚0 disk.sda.out‚ 652.6328125( proc.memory.availableMB‚ 337.18359375 network.recv‚ 4.083042e+06) disk./.usagePercent‚36.94428802953008 disk.dm-1.in‚ 0.0078125 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚97.94199466705322 cpu‚26.00452994139266 disk.sda2.out‚0$ disk./.usageGB‚550.4965896606445 disk.sda2.in‚0 disk.dm-1.out‚ 652.6328125 disk.rbd3.in‚0# disk.nvme0n1.in‚144999.02734375 disk.sda1.in‚0% disk.nvme0c0n1.in‚144999.02734375% disk.nvme0c0n1.out‚70699.53515625 disk.dm-0.out‚70699.53515625×SÖã‘Ô7‚(:† ¨èÒ„š¼¢ gpu.0.gpu‚96.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»¤B~Õ7‚(:” ¨èÒ›Îʤ disk.rbd1.in‚0.0625 disk.drbd2165.out‚0 disk.sda.out‚ 653.0234375 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.8997255e+07& disk.nvme0n1.out‚70714.50927734375 disk.rbd0.out‚0 cpu‚26.00425343312862 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 0.0078125# proc.memory.rssMB‚ 2808.91796875* proc.memory.percent‚17.144274711608887" disk.nvme0n1.in‚145007.7890625! disk.rbd1.out‚122.34423828125 disk.dm-0.in‚145007.7890625 network.recv‚ 4.086547e+06' proc.memory.availableMB‚ 336.9921875 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚145007.7890625( disk.nvme0c0n1.out‚70714.50927734375 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚70714.50927734375$ memory_percent‚97.94316291809082) disk./.usagePercent‚36.94425193246775$ disk./.usageGB‚550.4960517883301 disk.sda1.out‚0 disk.sda3.out‚ 653.0234375 disk.dm-1.out‚ 653.0234375ü„ 4Ö7/ _wandbruntime‚15450  _runtime‚15450U¡KÊ×7Ä & lr-AdamW/pg1‚9.778977110751223e-06 & lr-AdamW/pg2‚9.778977110751223e-06 trainer/global_step‚18199 % _timestamp‚1.7842882582709055e+09 _runtime‚15450.397353844 _step‚1497=a¿³ÏØ7É & lr-AdamW/pg1‚9.778977110751223e-06 & lr-AdamW/pg2‚9.778977110751223e-06 trainer/global_step‚18199 % _timestamp‚1.7842882582709055e+09 _runtime‚15450.397353844 _step‚1497Ù ý¯í÷4Ù7/ _wandbruntime‚15450  _runtime‚15450D)ï‡Ú7P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2785905168818176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144336230 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2766653062512640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2771642076580822 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19222720060374 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2752419339481088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842882582716901e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144336232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628706921577984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312928087 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚609243428579328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19251747927040 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39177394 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168591855 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144336232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19463492998656 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2752419356520448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 JADeviceStatsMonitor.on_train_batch_start/active.s  N€ mall_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2785904810439680 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2766653079552000 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2766653062512640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19252089266176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19251747927040 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168591855 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144336232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19463492998656 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2752419356520448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2785904810439680 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2766653079552000 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2766653062512640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19252089266176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19251747927040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Ú ¨-K4Ü7/ _wandbruntime‚15451  _runtime‚15451sÎüÃ'Ý7‹N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312944785 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2771789598306528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2766800320581632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71059491 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2786053064096256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168600921 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312934792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31880026 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19253101900288 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19252760553984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144343862 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168590930 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19464512990208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312934792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2766800303542272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842882588839147e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312944785 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144343864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚609276838725632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19223731114208 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2766800320581632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31880023 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚609276857337856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2752565867192320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2786053422481920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚18199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1499 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144343864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144343862 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19223390564206 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2752565850152960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39179465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19464416031744 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2786053422481920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39179180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚15451.009764717 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19253101900288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71059203 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2752565850152960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39179465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19464416031744 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2786053422481920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39179180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚15451.009764717 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19253101900288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71059203 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842882588839147e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312944785 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144343864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚609276838725632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19223731114208 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2766800320581632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31880023 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚609276857337856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2752565867192320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peZ†šE ak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2786053422481920Û \?%Î4ß7/ _wandbruntime‚15451  _runtime‚15451¤Æ 3à7ˆ 2loss/SRqqt1/assignment_loss‚0.4421473781877501 3loss/FBt1/assignment_loss‚0.0007474789363172461 3loss/FBt2/assignment_loss‚9.593138166318719e-05 /loss/FBt1/detection_loss‚0.3472154305896596 % _timestamp‚1.7842882588855307e+09 _step‚1500 0loss/FRt1/assignment_loss‚1.4994499787178814 /loss/FRt2/detection_loss‚1.2136346082117175 1loss/SRqqt1/detection_loss‚0.6715928002819033 1loss/SRqqt2/detection_loss‚0.6237873920886021 /loss/FBt2/detection_loss‚0.6493443638454968 %loss/total_loss‚7.218206042832739 .loss/FRt1/detection_loss‚0.495876502100744 trainer/global_step‚18199 0loss/FRt2/assignment_loss‚0.7980429633107615 2loss/SRqqt2/assignment_loss‚0.4762712151802424 epoch‚45 _runtime‚15451.010864825¨ûÖ*á7 2loss/SRqqt2/assignment_loss‚0.4762712151802424 epoch‚45 _runtime‚15451.010864825 2loss/SRqqt1/assignment_loss‚0.4421473781877501 3loss/FBt1/assignment_loss‚0.0007474789363172461 3loss/FBt2/assignment_loss‚9.593138166318719e-05 /loss/FBt1/detection_loss‚0.3472154305896596 % _timestamp‚1.7842882588855307e+09 _step‚1500 0loss/FRt1/assignment_loss‚1.4994499787178814 /loss/FRt2/detection_loss‚1.2136346082117175 1loss/SRqqt1/detection_loss‚0.6715928002819033 1loss/SRqqt2/detection_loss‚0.6237873920886021 /loss/FBt2/detection_loss‚0.6493443638454968 %loss/total_loss‚7.218206042832739 .loss/FRt1/detection_loss‚0.495876502100744 trainer/global_step‚18199 0loss/FRt2/assignment_loss‚0.7980429633107615Ü dH‡N›â7‚(: ¨èÒäÞ㣠gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚285.199' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.03980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uµÐ.#ã7‚(:˜ ¨èÒ‚Ï”¥ disk.sda2.out‚0$ memory_percent‚97.94623851776123 disk.sda2.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚122.34423828125# disk.dm-0.out‚70727.72216796875 disk.sda1.in‚0( proc.memory.availableMB‚ 336.48828125 network.recv‚ 4.090088e+06 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 disk.dm-0.in‚145023.25390625( disk.nvme0c0n1.out‚70727.72216796875 disk.dm-1.in‚ 0.0078125 disk.sda1.out‚0$ disk./.usageGB‚550.5022125244141 disk.dm-1.out‚ 653.1484375# proc.memory.rssMB‚ 2808.91796875) disk./.usagePercent‚36.94466538463555 disk.drbd2165.in‚0 disk.sda.out‚ 653.1484375 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625& disk.nvme0n1.out‚70727.72216796875* proc.memory.percent‚17.144274711608887 network.sent‚ 2.9002343e+07 disk.rbd0.in‚0 disk.sda3.out‚ 653.1484375 cpu‚26.00397326725076 proc.cpu.threads‚17£¢·‘ä7‚(:† Ÿ¨èÒíë³¢ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w?ùå7‚(:” Ÿ¨èÒÆ‹¥ disk.dm-0.in‚145023.25390625# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2808.91796875* disk./.usagePercent‚36.945016114957454# disk./.usageGB‚550.507438659668* proc.memory.percent‚17.144274711608887% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625 network.recv‚ 4.09432e+06 disk.dm-1.out‚ 653.453125 disk.rbd3.out‚0( proc.memory.availableMB‚ 336.73046875 cpu‚26.003712031489314 network.sent‚ 2.9067538e+07! disk.rbd1.out‚122.40673828125 disk.rbd0.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚70738.11962890625 disk.sda.out‚ 653.453125 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 653.453125# disk.dm-0.out‚70738.11962890625& disk.nvme0n1.out‚70738.11962890625 disk.sda1.in‚0# memory_percent‚97.9447603225708 disk.sda2.out‚0Ï|4æ7/ _wandbruntime‚15490  _runtime‚15490+[ÐOÈç7 % lr-AdamW/pg2‚9.24705950928931e-06 trainer/global_step‚18249 % _timestamp‚1.7842882980427074e+09 _runtime‚15490.168410226 _step‚1501 % lr-AdamW/pg1‚9.24705950928931e-06ì==©Íè7Ç trainer/global_step‚18249 % _timestamp‚1.7842882980427074e+09 _runtime‚15490.168410226 _step‚1501 % lr-AdamW/pg1‚9.24705950928931e-06 % lr-AdamW/pg2‚9.24705950928931e-06Ý üW¦€4é7/ _wandbruntime‚15490  _runtime‚15490SëI(ê7‚P BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2793317915801600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2793317915801600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19302784768000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313764139 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144717830 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144717832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169046307 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19302443428864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39281321 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19272995997392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144717830 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31965528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2774015131033600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2779017869072080 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169046307 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19514620258816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842882980438607e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 _step‚1502 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2774015113994240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71246852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31965531 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2793317557423104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚610914847893504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2793317557423104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71246583 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610914866505728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 _runtime‚15490.169648614 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19302443428864ϸûc (ë7‡P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2793317915801600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2793317915801600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19302784768000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313764139 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144717830 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144717832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169046307 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19302443428864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39281321 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19272995997392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144717830 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31965528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2774015131033600 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2779017869072080 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169046307 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19514620258816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7842882980438607e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 _step‚1502 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2774015113994240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71246852 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31965531 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2793317557423104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚610914847893504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2793317557423104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71246583 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610914866505728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 _runtime‚15490.169648614 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19302443428864 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313780837 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169055373 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610948276652032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7842882986562638e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144725464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610948295264256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2779165748387036 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19274347601116 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2774162355023872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19303456055808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚ 15490.7821316 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144725462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39283409 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2774162372063232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31967273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71250685 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚630463820443136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2774162355023872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313770844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313780837 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19515636555264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2793466169465344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 16904538^ Œ&’ 2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71250397 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39283124 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚630463931819520 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2759891400785920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144725462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31967276 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2774162372063232 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19303797402112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144725464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2759891383746560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664"ï²J'î7ŽN U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2793466169465344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169045382 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71250397 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39283124 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚630463931819520 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2759891400785920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144725462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31967276 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2774162372063232 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19303797402112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144725464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2759891383746560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169055373 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2793466169465344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19303797402112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313780837 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169055373 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610948276652032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7842882986562638e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144725464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610948295264256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2779165748387036 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19274347601116 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2774162355023872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19303456055808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚ 15490.7821316 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144725462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39283409 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2774162372063232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31967273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71250685 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚630463820443136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2774162355023872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313770844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313780837 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19515636555264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ß Ut³Q4ï7/ _wandbruntime‚15490  _runtime‚15490èódPð7‹ 3loss/FBt1/assignment_loss‚0.0009696309021529308 /loss/FRt1/detection_loss‚0.5148907486200333 1loss/SRqqt2/detection_loss‚0.5747225353563071 /loss/FBt1/detection_loss‚0.4490832227326919 %loss/total_loss‚6.764277526198245 _runtime‚15490.784686512 _step‚1504 0loss/FRt2/assignment_loss‚0.7140608912875467 2loss/SRqqt1/assignment_loss‚0.3865049423520195 /loss/FRt2/detection_loss‚1.2263003754261965 epoch‚45 trainer/global_step‚18249 % _timestamp‚1.7842882986593583e+09 0loss/FRt1/assignment_loss‚1.4337113714709724 4loss/FBt2/assignment_loss‚5.7641037768161956e-05 3loss/SRqqt2/assignment_loss‚0.44555218968188254 1loss/SRqqt1/detection_loss‚0.7771862185529379 /loss/FBt2/detection_loss‚0.2412377587777363Ãe}ñ7 3loss/SRqqt2/assignment_loss‚0.44555218968188254 1loss/SRqqt1/detection_loss‚0.7771862185529379 /loss/FBt2/detection_loss‚0.2412377587777363 3loss/FBt1/assignment_loss‚0.0009696309021529308 /loss/FRt1/detection_loss‚0.5148907486200333 1loss/SRqqt2/detection_loss‚0.5747225353563071 /loss/FBt1/detection_loss‚0.4490832227326919 %loss/total_loss‚6.764277526198245 _runtime‚15490.784686512 _step‚1504 0loss/FRt2/assignment_loss‚0.7140608912875467 2loss/SRqqt1/assignment_loss‚0.3865049423520195 /loss/FRt2/detection_loss‚1.2263003754261965 epoch‚45 trainer/global_step‚18249 % _timestamp‚1.7842882986593583e+09 0loss/FRt1/assignment_loss‚1.4337113714709724 4loss/FBt2/assignment_loss‚5.7641037768161956e-05à —ˆ„žò7‚(:“ ®¨èÒ›ÎÇ¢ gpu.0.gpu‚92.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚248.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.782199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç™ä>#ó7‚(:˜ ®¨èÒìÉê¥ disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 653.7890625# disk.dm-0.out‚70750.10400390625 proc.cpu.threads‚17# disk./.usageGB‚550.512752532959! disk.rbd1.out‚122.40673828125 disk.drbd2165.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625 disk.rbd1.in‚0.0625 disk.sda.out‚ 653.7890625 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚97.94509410858154 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625& disk.nvme0n1.out‚70750.10400390625( disk.nvme0c0n1.out‚70750.10400390625 disk.dm-1.in‚ 0.0078125 network.recv‚ 4.097697e+06 disk.sda2.out‚0 disk.dm-0.in‚145023.25390625* proc.memory.percent‚17.144274711608887 disk.dm-1.out‚ 653.7890625 disk.sda1.out‚0( proc.memory.availableMB‚ 336.67578125 cpu‚26.003400255632627) disk./.usagePercent‚36.94537273345266 disk.drbd2165.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2808.91796875 network.sent‚ 2.9070502e+07 disk.sda.in‚ 0.0078125‰ûˆ;“ô7‚(:ˆ ½¨èÒȦ٢ gpu.0.gpu‚88.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë¦àý"õ7‚(:— ½¨èÒÃÜæ¤ disk.sda3.out‚ 654.0390625( proc.memory.availableMB‚ 336.64453125 network.recv‚ 4.102285e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125" disk.dm-0.out‚70760.3310546875 disk.sda1.in‚0 disk.sda.out‚ 654.0390625* proc.memory.percent‚17.144274711608887 disk.dm-1.in‚ 0.0078125 proc.cpu.threads‚17! disk.rbd1.out‚122.46923828125 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125# proc.memory.rssMB‚ 2808.91796875 cpu‚26.003143438597125% disk.nvme0n1.out‚70760.3310546875 disk.sda1.out‚0% disk.nvme0c0n1.in‚145023.25390625 network.sent‚ 2.9133679e+07 disk.dm-1.out‚ 654.0390625 disk.rbd1.in‚0.0625* disk./.usagePercent‚36.945798985997165 disk.rbd3.out‚0 disk.dm-0.in‚145023.25390625 disk.drbd2165.in‚0$ memory_percent‚97.94528484344482# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚70760.3310546875$ disk./.usageGB‚550.5191040039062ùîö7‚(:’ ̨è񠕢 gpu.0.gpu‚77.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.057' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.61140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$_‰m÷7‚(: ̨èÒÎ×å¤* disk./.usagePercent‚36.946184277336194 cpu‚26.002867431170696 network.sent‚ 2.9138827e+07# disk./.usageGB‚550.524845123291 disk.rbd3.in‚0 disk.sda.in‚ 0.0078125 disk.rbd1.in‚0.0625 network.recv‚ 4.105595e+06 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625# disk.nvme0n1.out‚70772.76171875 disk.sda2.out‚0$ memory_percent‚97.94528484344482 disk.dm-0.in‚145023.25390625 disk.rbd0.out‚0 disk.sda.out‚ 654.265625 disk.sda3.out‚ 654.265625 disk.dm-1.out‚ 654.265625( proc.memory.availableMB‚ 336.64453125# proc.memory.rssMB‚ 2808.91796875 proc.cpu.threads‚17 disk.dm-0.out‚70772.76171875 disk.sda1.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625! disk.rbd1.out‚122.46923828125 disk.sda1.in‚0% disk.nvme0c0n1.out‚70772.76171875 disk.dm-1.in‚ 0.0078125* proc.memory.percent‚17.144274711608887 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.0078125}"!¨4ø7/ _wandbruntime‚15530  _runtime‚15530ÿC¯RÊù7Ä & lr-AdamW/pg1‚8.729727284538578e-06 & lr-AdamW/pg2‚8.729727284538578e-06 trainer/global_step‚18299 % _timestamp‚1.7842883380365014e+09 _runtime‚15530.162050301 _step‚1505à²qNÏú7É & lr-AdamW/pg1‚8.729727284538578e-06 & lr-AdamW/pg2‚8.729727284538578e-06 trainer/global_step‚18299 % _timestamp‚1.7842883380365014e+09 _runtime‚15530.162050301 _step‚1505á ‰0-4û7/ _wandbruntime‚15530  _runtime‚15530ûÞ!(ü7€P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145099430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚612586304432128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2781377182515200 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2781377165475840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚612586285819904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19323953034190 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145099432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2781377182515200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71437727 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19565693742080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2800730304406528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71437990 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2786394376741838 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2786394019152588 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 _step‚1506 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2800730304406528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169490782 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39385209 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚632151998174208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19353480269824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2767070406668288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145099432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19565605165056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842883380375044e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 314590212 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32052778 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19353138930688O*’V ý7…P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `MDeviceStatøãÐXµsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19565605165056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842883380375044e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 314590212 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32052778 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19353138930688 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145099430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚612586304432128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2781377182515200 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2781377165475840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚612586285819904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19323953034190 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145099432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2781377182515200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71437727 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19565693742080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2800730304406528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71437990 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2786394376741838 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2786394019152588 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 _step‚1506 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2800730304406528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169490782 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39385209 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚632151998174208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19353480269824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2767070406668288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145099432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101â Ô4y4þ7/ _wandbruntime‚15530  _runtime‚15530p x;'ÿ7‹N E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145107064 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2781524406505472 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2800878558063104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚632186346115584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2767216917340160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 314606896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19566631537152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19324964088024 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314616889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚612619714578432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚1507 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19354151557632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145107064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39387300 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2767216934379520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚612619733190656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2800878916448768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32054526 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19324623538022 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71441565 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19354492903936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2786541898467544 % _timestamp‚1.7842883386478865e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2800878916448768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32054526 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19324623538022 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71441565 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19354492903936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2786541898467544 % _timestamp‚1.7842883386478865e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19324964088024 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314616889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚612619714578432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚1507 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19354151557632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145107064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39387300 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2767216934379520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚612619733190656ã ‰ÌJ<48/ _wandbruntime‚15530  _runtime‚15530ð‘…a‚8Š %loss/total_loss‚6.670317494553925 _runtime‚15530.776297896 _step‚1508 epoch‚45 3loss/SRqqt1/assignment_loss‚0.39810196721364843 2loss/SRqqt2/assignment_loss‚0.3830024239876153 3loss/FBt1/assignment_loss‚0.0004787387759414089 /loss/FRt2/detection_loss‚1.1953519786381794 1loss/SRqqt1/detection_loss‚0.7063543174737892 0loss/SRqqt2/detection_loss‚0.539080968210374 % _timestamp‚1.7842883386509137e+09 0loss/FRt2/assignment_loss‚0.8267776370048523 3loss/FBt2/assignment_loss‚0.0002815072124276087 /loss/FRt1/detection_loss‚0.4613067967616129 0loss/FBt2/detection_loss‚0.36096031829698144 0loss/FRt1/assignment_loss‚1.5074990530896293 /loss/FBt1/detection_loss‚0.2911217878888737 trainer/global_step‚18299@ã%ƒ8 %loss/total_loss‚6.670317494553925 _runtime‚15530.776297896 _step‚1508 epoch‚45 3loss/SRqqt1/assignment_loss‚0.39810196721364843 2loss/SRqqt2/assignment_loss‚0.3830024239876153 3loss/FBt1/assignment_loss‚0.0004787387759414089 /loss/FRt2/detection_loss‚1.1953519786381794 1loss/SRqqt1/detection_loss‚0.7063543174737892 0loss/SRqqt2/detection_loss‚0.539080968210374 % _timestamp‚1.7842883386509137e+09 0loss/FRt2/assignment_loss‚0.8267776370048523 3loss/FBt2/assignment_loss‚0.0002815072124276087 /loss/FRt1/detection_loss‚0.4613067967616129 0loss/FBt2/detection_loss‚0.36096031829698144 0loss/FRt1/assignment_loss‚1.5074990530896293 /loss/FBt1/detection_loss‚0.2911217878888737 trainer/global_step‚18299ä „4„„8‚(:’ Û¨èÒŠì £ gpu.0.gpu‚53.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.30499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ç~$…8‚(:™ Û¨èÒãÂÛ¤ network.recv‚ 4.109215e+06) disk./.usagePercent‚36.94604833733552# disk.dm-0.out‚70784.29248046875' proc.memory.availableMB‚ 336.3828125# proc.memory.rssMB‚ 2808.91796875* proc.memory.percent‚17.144274711608887 cpu‚26.002592269073077 network.sent‚ 2.9143909e+07# disk./.usageGB‚550.522819519043 disk.sda1.out‚0 disk.sda.out‚ 654.54296875 disk.sda2.out‚0& disk.nvme0n1.out‚70784.29248046875 disk.sda2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚122.53173828125 disk.dm-1.out‚ 654.54296875( disk.nvme0c0n1.out‚70784.29248046875 disk.sda3.out‚ 654.54296875 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚145023.25390625# memory_percent‚97.9468822479248¨µ@ “†8‚(:ˆ ê¨èÒèϾ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"žm‡8‚(:‘ ê¨èÒÿœþ¤ disk.sda1.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda.out‚ 654.703125* proc.memory.percent‚17.144274711608887# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚70797.6455078125 disk.sda2.in‚0 cpu‚26.00230036999303" disk.dm-0.out‚70797.6455078125% disk.nvme0n1.out‚70797.6455078125 disk.sda.in‚ 0.0078125 disk.rbd3.in‚0' proc.memory.availableMB‚ 336.6015625# proc.memory.rssMB‚ 2808.91796875$ disk./.usageGB‚550.5276679992676 disk.dm-0.in‚145023.25390625 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0078125$ memory_percent‚97.94554710388184 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda3.out‚ 654.703125 disk.dm-1.out‚ 654.703125! disk.rbd1.out‚122.53173828125 network.sent‚ 2.9206062e+07 network.recv‚ 4.112673e+06) disk./.usagePercent‚36.94637372291154 disk.rbd0.out‚0 disk.rbd1.in‚0.0625„æÓˆ8‚(:ˆ ù¨èÒÁÓ¢¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gO¹$QÀpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ¸‰8‚(:‘ ù¨èÒ•ª§¤ disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.144274711608887& disk.nvme0c0n1.out‚70806.404296875 disk.sda.in‚ 0.0078125$ memory_percent‚97.94554710388184 network.sent‚ 2.9211265e+07 network.recv‚ 4.116316e+06$ disk./.usageGB‚550.5329666137695 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125$ disk.nvme0n1.out‚70806.404296875 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0# proc.memory.rssMB‚ 2808.91796875 cpu‚26.002028369783094( disk./.usagePercent‚36.9467293173766# disk.nvme0n1.in‚145023.25390625% disk.nvme0c0n1.in‚145023.25390625 proc.cpu.threads‚17 disk.dm-0.in‚145023.25390625! disk.dm-0.out‚70806.404296875 disk.drbd2165.out‚0 disk.sda3.out‚ 654.9296875! disk.rbd1.out‚122.59423828125' proc.memory.availableMB‚ 336.6015625 disk.drbd2165.in‚0 disk.sda.out‚ 654.9296875 disk.dm-1.out‚ 654.9296875¤ ÷f4Š8/ _wandbruntime‚15570  _runtime‚15570À ÙÉ‹8à & lr-AdamW/pg1‚8.227013663781148e-06 & lr-AdamW/pg2‚8.227013663781148e-06 trainer/global_step‚18349 % _timestamp‚1.7842883780075788e+09 _runtime‚15570.13318898 _step‚15094Ä´ ÎŒ8È & lr-AdamW/pg1‚8.227013663781148e-06 & lr-AdamW/pg2‚8.227013663781148e-06 trainer/global_step‚18349 % _timestamp‚1.7842883780075788e+09 _runtime‚15570.13318898 _step‚1509å žÛÜ}48/ _wandbruntime‚15570  _runtime‚15570ᦦ(Ž8P JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169955211 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2788739216957440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2788739233996800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚614257723746304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2793770169233096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2774395940261888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2808143051389952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚15570.137383878 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315436243 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169945234 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71628688 _step‚1510 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32140031 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚633874420013568 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145481032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145481030 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1"§p# (8†P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315436243 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169945234 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71628688 _step‚1510 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32140031 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚633874420013568 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145481032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145481030 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169955211 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2788739216957440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2788739233996800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚614257723746304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2793770169233096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2774395940261888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2808143051389952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚15570.137383878 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088æ åhMÛ48/ _wandbruntime‚15570  _runtime‚15570^÷ƒ'‘8‰N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39490378 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 # _timestamp‚1.78428837862082e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19617711407616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145488664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32141773 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19405188405760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71632439 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2788886475026432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2808291663432192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39490663 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19375580574932 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169954286 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2808291663432192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2788886475026432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169964277 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145488662 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315442948 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71632151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32141776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315452941 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2788886457987072 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2774542450933760 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2808291305046528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19404847059456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19375240024930 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315452941 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19404847059456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2774542467973120 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145488662 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚633908863912448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚18349 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2788886457987072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚614291171117056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169964277 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚1511 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438tø #‚’8ŽN T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2808291305046528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19404847059456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19375240024930 JBDeviceStatsMonitor.onzi#ä’%_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315452941 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19404847059456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2774542467973120 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145488662 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚633908863912448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚18349 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2788886457987072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚614291171117056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169964277 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚1511 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39490378 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 # _timestamp‚1.78428837862082e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19617711407616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145488664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32141773 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19405188405760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71632439 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2788886475026432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2808291663432192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39490663 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19375580574932 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169954286 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2808291663432192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2788886475026432 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169964277 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145488662 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315442948 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71632151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32141776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315452941 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2788886457987072 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2774542450933760ç z^4“8/ _wandbruntime‚15570  _runtime‚15570ZS&[”8ˆ trainer/global_step‚18349 2loss/FBt2/assignment_loss‚0.002232578750994574 /loss/FRt1/detection_loss‚0.4991967027262985 /loss/FRt2/detection_loss‚1.1665113668591722 _runtime‚15570.748709642 _step‚1512 0loss/FRt1/assignment_loss‚1.5027151707817026 2loss/SRqqt2/assignment_loss‚0.4306610999099874 0loss/FBt1/detection_loss‚0.40981519039983044 /loss/FBt2/detection_loss‚0.4014259080718001 2loss/FBt1/assignment_loss‚0.010296367023491668 1loss/SRqqt1/detection_loss‚0.7731272236175839 %loss/total_loss‚6.889814189031624 % _timestamp‚1.7842883786233666e+09 0loss/FRt2/assignment_loss‚0.7377806676315589 2loss/SRqqt1/assignment_loss‚0.4040814891650611 1loss/SRqqt2/detection_loss‚0.5519704240941422 epoch‚45×4o•8 trainer/global_step‚18349 2loss/FBt2/assignment_loss‚0.002232578750994574 /loss/FRt1/detection_loss‚0.4991967027262985 /loss/FRt2/detection_loss‚1.1665113668591722 _runtime‚15570.748709642 _step‚1512 0loss/FRt1/assignment_loss‚1.5027151707817026 2loss/SRqqt2/assignment_loss‚0.4306610999099874 0loss/FBt1/detection_loss‚0.40981519039983044 /loss/FBt2/detection_loss‚0.4014259080718001 2loss/FBt1/assignment_loss‚0.010296367023491668 1loss/SRqqt1/detection_loss‚0.7731272236175839 %loss/total_loss‚6.889814189031624 % _timestamp‚1.7842883786233666e+09 0loss/FRt2/assignment_loss‚0.7377806676315589 2loss/SRqqt1/assignment_loss‚0.4040814891650611 1loss/SRqqt2/detection_loss‚0.5519704240941422 epoch‚45è ‚K¼jž–8‚(:“ ˆ©èÒ·µÛ£ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.464800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ààþu—8‚(:’ ˆ©èÒû¶ÿ¤ disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.sda.out‚ 655.1640625 network.recv‚ 4.119319e+06) disk./.usagePercent‚36.94746380299233! disk.dm-0.out‚70817.982421875 disk.sda1.in‚0 disk.dm-1.in‚ 0.0078125 network.sent‚ 2.9216192e+07$ disk./.usageGB‚550.5439109802246% disk.nvme0c0n1.in‚145023.25390625 disk.sda2.in‚0 cpu‚26.001768109799325 disk.dm-0.in‚145023.25390625 disk.sda1.out‚0& disk.nvme0c0n1.out‚70817.982421875 disk.sda2.out‚0 disk.sda3.out‚ 655.1640625 disk.rbd3.in‚0# disk.nvme0n1.in‚145023.25390625 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125! disk.rbd1.out‚122.59423828125* proc.memory.percent‚17.144274711608887 disk.dm-1.out‚ 655.1640625# proc.memory.rssMB‚ 2808.91796875 proc.cpu.threads‚17$ disk.nvme0n1.out‚70817.982421875 disk.drbd2165.in‚0 disk.rbd0.in‚0# memory_percent‚97.9454755783081( proc.memory.availableMB‚ 336.61328125¸D©à“˜8‚(:ˆ —©èÒêìÄ¢ gpu.0.gpu‚83.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÃû%™8‚(:š —©èÒå£Ü¤ disk.sda.out‚ 655.49609375 disk.sda2.in‚0 disk.dm-0.in‚145023.25390625$ memory_percent‚97.94304370880127 cpu‚26.00151170905761# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.out‚0! disk.rbd1.out‚122.65673828125 network.recv‚ 4.123871e+06 disk.dm-1.in‚ 0.0078125* proc.memory.percent‚17.144274711608887& disk.nvme0n1.out‚70827.67822265625 disk.rbd3.in‚0( proc.memory.availableMB‚ 337.01171875 network.sent‚ 2.927929e+07) disk./.usagePercent‚36.94793792894575% disk.nvme0c0n1.in‚145023.25390625( disk.nvme0c0n1.out‚70827.67822265625 disk.sda3.out‚ 655.49609375 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2808.91796875$ disk./.usageGB‚550.5509757995605 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 655.49609375 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0# disk.dm-0.out‚70827.67822265625ðÌî4š8/ _wandbruntime‚15609  _runtime‚15609ÞKõÊ›8Ä % _timestamp‚1.7842884176784422e+09 _runtime‚15609.804014805 _step‚1513 & lr-AdamW/pg1‚7.738950935373501e-06 & lr-AdamW/pg2‚7.738950935373501e-06 trainer/global_step‚183991Ÿ¦Ïœ8É & lr-AdamW/pg2‚7.738950935373501e-06 trainer/global_step‚18399 % _timestamp‚1.7842884176784422e+09 _runtime‚15609.804014805 _step‚1513 & lr-AdamW/pg1‚7.738950935373501e-06é ÊÇçõ48/ _wandbruntime‚15609  _runtime‚15609é%~µ(ž8P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145862632 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32227278 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2796101268439040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145862630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71818444 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170399686 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170409663 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170399686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145862632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316272295 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 % _timestamp‚1.7842884176794446e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2796101268439040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2815555798373376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚15609.808376968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316262316 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19454871273472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19454529934336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39591166 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2801146676902854 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2781721473855488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚18399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19667773656064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2781721490894848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19425186008006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170409663 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71818708 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635597042518016 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2801146319313604 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2815556156751872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2815556156751872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2796101285478400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2796101285478400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19454529934336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77ðâ<Ÿ8†P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316272295 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 % _timestamp‚1.7842884176794446e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2796101268439040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2815555798373376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚15609.808376968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316262316 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19454871273472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19454529934336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39591166 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2801146676902854 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2781721473855488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚18399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19667773656064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2781721490894848 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19425186008006 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170409663 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71818708 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635597042518016 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2801146319313604 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2815556156751872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2815556156751872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2796101285478400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2796101285478400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19454529934336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145862632 G>DeviceStatsMonõ”¸JÐ itor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32227278 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2796101268439040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145862630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71818444 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170399686 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170409663 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170399686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145862632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376ê ŠYµ4 8/ _wandbruntime‚15610  _runtime‚15610Á ­'¡8‹N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635631382914560 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170408738 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _runtime‚15610.417505083 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2815704410415616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145870262 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71822198 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19455542561280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316288993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚635631492193792 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170418729 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19668883150336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2781867984527360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19455542561280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316279000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39593175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19455883907584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚615962590431232 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2801294198628560 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2781868001566720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316279000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32229026 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145870264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2815704052029952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145870264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19455883907584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145870262 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19668792483328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2815704410415616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19426197061840 trainer/global_step‚18399 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19455883907584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145870262 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19668792483328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2815704410415616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19426197061840 trainer/global_step‚18399 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _runtime‚15610.417505083 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2815704410415616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145870262 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71822198 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19455542561280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316288993 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚635631492193792 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170418729 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19668883150336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2781867984527360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19455542561280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316279000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39593175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19455883907584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚615962590431232 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2801294198628560 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2781868001566720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316279000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32229026 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145870264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2815704052029952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145870264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B‰×l?©8–{ =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3295484185218811 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6605631113052368 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21653014421463013 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.612106442451477 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 particle/accuracy_2_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.570231556892395 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6221827864646912 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.643156886100769 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1976667195558548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 )loss/val_total_loss‚8.087331160739282 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481790542603 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.355757474899292 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.57386314868927 +jet/accuracy_2_of_2‚0.15857461094856262 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8526139855384827 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6251385807991028 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7611375451087952 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2688971161842346 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5899950861930847 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6010287404060364 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9134987592697144 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6318121552467346 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7883504033088684 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26265454292297363 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23675616085529327 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7755644917488098 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22400036454200745 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5952374339103699 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8016327619552612 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7593721747398376 particle/accuracy_3_of_4‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6265944242477417 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021314218640327454 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 trainer/global_step‚18399 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6282411813735962 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04022018983960152 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 epoch‚45 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3749952018260956 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6906125545501709 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07653598487377167 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8716074824333191 *jet/accuracy_2_of_4‚0.8048639893531799 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4511719346046448 'particle/f_score‚0.6248348951339722 jet/accuracy_4_of_5‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6010287404060364 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021704116836190224 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23764507472515106 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8716074824333191 *jet/accuracy_1_of_3‚0.9243618845939636 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6219042539596558 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6593256592750549 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017985131591558456 % _timestamp‚1.7842884355357075e+09 _step‚1517 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2425893098115921 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2675999104976654 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 _runtime‚15627.661923591 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02117072604596615 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 +particle/sensitivity‚0.6147289276123047 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35613980889320374 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31941133737564087 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.146763nÓ]"3992433548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6609604358673096 jet/accuracy_3_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02185741253197193 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22410427033901215 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.05690142139792442 *jet/accuracy_1_of_4‚0.9830800294876099 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017985131591558456 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.261829137802124 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045978933572769165 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25435179471969604 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5819154381752014 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 particle/accuracy_2_of_4‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20614655315876007 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2839868366718292 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7148526310920715 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7334467172622681 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.541688084602356 +jet/accuracy_1_of_1‚0.32793644070625305 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035846542567014694 +jet/accuracy_3_of_4‚0.07530007511377335 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24795766174793243 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27327802777290344 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04365329071879387 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20921795070171356 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23829078674316406 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 particle/accuracy_3_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3716375231742859 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6597731113433838 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25389179587364197 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6552693247795105 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20154103636741638 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9035035371780396 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7684324979782104 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5939751267433167 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7219009399414062 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5286114811897278 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5947253704071045 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5294855833053589 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7953447699546814 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 *jet/accuracy_2_of_3‚0.5737629532814026 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995073139667511 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2720049321651459 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6562471389770508 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29547354578971863 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 (particle/accuracy‚0.8414868712425232 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08774648606777191 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970898389816284 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5049999952316284 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6532770991325378 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6208987832069397 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6241739392280579 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9490876197814941 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886239171028137 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_4‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49235597252845764 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.261829137802124 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7225654125213623 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942677617073059 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481790542603 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7289450764656067 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2160274237394333 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27180013060569763 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857453107833862 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23807403445243835 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5996026396751404 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6940297484397888 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8468281030654907 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7200394868850708 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24919645488262177 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5449755787849426 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6138482093811035 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7177626490592957 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1€ ¡=ª8›{ =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 +particle/sensitivity‚0.6147289276123047 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35613980889320374 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31941133737564087 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6609604358673096 jet/accuracy_3_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02185741253197193 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22410427033901215 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.05690142139792442 *jet/accuracy_1_of_4‚0.9830800294876099 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017985131591558456 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.261829137802124 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045978933572769165 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25435179471969604 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5819154381752014 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 particle/accuracy_2_of_4‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20614655315876007 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2839868366718292 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7148526310920715 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7334467172622681 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.541688084602356 +jet/accuracy_1_of_1‚0.32793644070625305 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035846542567014694 +jet/accuracy_3_of_4‚0.07530007511377335 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24795766174793243 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27327802777290344 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04365329071879387 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20921795070171356 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23829078674316406 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 particle/accuracy_3_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3716375231742859 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6597731113433838 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25389179587364197 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6552693247795105 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20154103636741638 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9035035371780396 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7684324979782104 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5939751267433167 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7219009399414062 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5286114811897278 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5947253704071045 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5294855833053589 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7953447699546814 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 *jet/accuracy_2_of_3‚0.5737629532814026 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995073139667511 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2720049321651459 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6562471389770508 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29547354578971863 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 (particle/accuracy‚0.8414868712425232 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08774648606777191 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970898389816284 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5049999952316284 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6532770991325378 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6208987832069397 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6241739392280579 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9490876197814941 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886239171028137 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_4‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49235597252845764 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.261829137802124 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7225654125213623 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942677617073059 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481790542603 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7289450764656067 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2160274237394333 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27180013060569763 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857453107833862 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23807403445243835 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5996026396751404 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6940297484397888 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8468281030654907 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7200394868850708 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24919645488262177 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5449755787849426 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6138482093811035 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7177626490592957 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3295484185218811 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6605631113052368 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21653014421463013 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.612106442451477 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 particle/accuracy_2_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.570231556892395 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6221827864646912 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.643156886100769 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1976667195558548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 )loss/val_total_loss‚8.087331160739282 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481790542603 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.355757474899292 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.57386314868927 +jet/accuracy_2_of_2‚0.15857461094856262 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8526139855384827 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6251385807991028 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7611375451087952 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2688971161842346 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5899950861930847 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6010287404060364 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9134987592697144 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6318121552467346 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7883504033088684 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26265454292297363 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23675616085529327 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7755644917488098 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22400036454200745 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5952374339103699 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8016327619552612 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7593721747398376 particle/accuracy_3_of_4‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6265944242477417 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.021314218640327454 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 trainer/global_step‚18399 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6282411813735962 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04022018983960152 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 epoch‚45 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3749952018260956 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6906125545501709 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07653598487377167 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8716074824333191 *jet/accuracy_2_of_4‚0.8048639893531799 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4511719346046448 'particle/f_score‚0.6248348951339722 jet/accuracy_4_of_5‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6010287404060364 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.021704116836190224 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23764507472515106 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8716074824333191 *jet/accuracy_1_of_3‚0.9243618845939636 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6219042539596558 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6593256592750549 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017985131591558456 % _timestamp‚1.7842884355357075e+09 _step‚1517 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2425893098115921 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2675999104976654 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 _runtime‚15627.661923591 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02117072604596615 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043í í¨[…“«8‚(:ˆ µ©èÒÅøž£ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ž Þ$¬8‚(:™ µ©èÒÅ‹š¥$ disk./.usageGB‚550.5699157714844 disk.rbd0.out‚0* proc.memory.percent‚17.148280143737793 network.recv‚ 4.132001e+06) disk./.usagePercent‚36.94920900635324 disk.sda1.out‚0' disk.nvme0c0n1.out‚70850.4580078125# proc.memory.rssMB‚ 2809.57421875# disk.nvme0n1.in‚145023.25390625% disk.nvme0n1.out‚70850.4580078125 disk.sda1.in‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda2.in‚0 disk.dm-1.out‚ 656.01953125 disk.rbd1.in‚0.0625 disk.rbd0.in‚0 disk.sda3.out‚ 656.01953125! disk.rbd1.out‚122.78173828125 disk.sda.out‚ 656.01953125 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.dm-0.in‚145023.25390625" disk.dm-0.out‚70850.4580078125( proc.memory.availableMB‚ 335.71484375 disk.sda.in‚ 0.0078125 cpu‚26.005270965483813 disk.drbd2165.in‚0 disk.sda2.out‚0 network.sent‚ 2.9347551e+07 disk.rbd3.out‚0$ memory_percent‚97.95095920562744 proc.cpu.threads‚17i/Г­8‚(:ˆ Ä©èÒ©÷«¢ gpu.0.gpu‚51.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öQÊÇ&®8‚(:› Ä©èÒçÿï¤ disk.sda3.out‚ 658.22265625 disk.dm-0.in‚145023.25390625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 335.7109375 disk.sda2.in‚0 disk.dm-1.in‚ 0.0078125 disk.rbd0.in‚0 network.sent‚ 2.9404841e+07% disk.nvme0c0n1.in‚145023.25390625( disk.nvme0c0n1.out‚70874.10595703125 disk.sda3.in‚ 0.0078125# disk.dm-0.out‚70874.10595703125$ disk./.usageGB‚550.5802192687988 disk.rbd1.in‚0.0625 disk.sda1.in‚0# proc.memory.rssMB‚ 2809.57421875* proc.memory.percent‚17.148280143737793 disk.dm-1.out‚ 658.22265625 disk.sda.in‚ 0.0078125 proc.cpu.threads‚17 disk.sda.out‚ 658.22265625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625& disk.nvme0n1.out‚70874.10595703125 disk.drbd2165.in‚0 network.recv‚ 4.135871e+06) disk./.usagePercent‚36.94990048270323! disk.rbd1.out‚122.78173828125$ memory_percent‚97.95098304748535 cpu‚26.005012130536954¯¹T¯8‚(:’ Ó©èÒ¤‰Í¢ gpu.0.gpu‚51.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚263.184' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.63680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËC˜$°8‚(:Ž Ó©èÒÏ”½¤' proc.memory.availableMB‚ 335.7109375 disk.dm-0.in‚145023.25390625 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚70884.8037109375# disk.nvme0n1.in‚145023.25390625 disk.sda1.out‚0 disk.sda.in‚ 0.0078125 disk.sda2.in‚0# proc.memory.rssMB‚ 2809.57421875 cpu‚26.004738136246626 proc.cpu.threads‚17 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚145023.25390625) disk./.usagePercent‚36.95014445788336 disk.sda1.in‚0 disk.sda2.out‚0! disk.rbd1.out‚122.84423828125* proc.memory.percent‚17.148280143737793% disk.nvme0n1.out‚70884.8037109375' disk.nvme0c0n1.out‚70884.8037109375 disk.sda3.out‚ 659.59375 disk.rbd1.in‚0.0625$ memory_percent‚97.95098304748535 network.sent‚ 2.9409925e+07 network.recv‚ 4.139488e+06 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 659.59375# disk./.usageGB‚550.583854675293 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 659.59375 disk.rbd3.out‚0žYˆ·4±8/ _wandbruntime‚15667  _runtime‚15667^Tïʲ8Ä & lr-AdamW/pg1‚7.265570446672658e-06 & lr-AdamW/pg2‚7.265570446672658e-06 trainer/global_step‚18449 % _timestamp‚1.7842884749015708e+09 _runtime‚15667.027155266 _step‚1518™§Ï³8É & lr-AdamW/pg1‚7.265570446672658e-06 & lr-AdamW/pg2‚7.265570446672658e-06 trainer/global_step‚18449 % _timestamp‚1.7842884749015708e+09 _runtime‚15667.027155266 _step‚1518î Ãh¯ã4´8/ _wandbruntime‚15667  _runtime‚15667¬š?m€ µ8€P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32317048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚637353630873600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚617616530311168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19493522220143 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19737214043136 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2824915845212672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2824916203591168 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2805392886571008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32317051 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19737119174656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2805392903610368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39727163 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19522958641664 aMDeviceStatsMonitor.on_train_batch_start/ë}“†inactive_split_bytes.large_pool.freed‚617616511698944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146364898 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170924460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2810448852333421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1519 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2824916203591168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚18449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317289356 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2824915845212672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2790955687702528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚15667.028515984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19523299980800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146364896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19522958641664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19523299980800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19493181670253 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72043939 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚637353744354304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2824915845212672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2824916203591168 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2805392886571008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32317051 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19737119174656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2805392903610368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39727163 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19522958641664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617616511698944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146364898 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170924460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2810448852333421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1519 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2824916203591168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚18449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317289356 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2824915845212672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2790955687702528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚15667.028515984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19523299980800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146364896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19522958641664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19523299980800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19493181670253 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72043939 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚637353744354304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19738147664384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19523971268608 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2825064457254912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚637388088121856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2805540127600640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170943503 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39728965 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170943503 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19524312614912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19494533273977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39729230 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2791102181335040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146372528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2791102198374400 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚18449 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2825064098869248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2805540144640000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146372530 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72048026 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32318796 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _runtime‚15667.640533913 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317316033 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32318793 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170933512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72047758 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146372528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19523971268608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617649940457472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19738232039936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2825064457254912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146372530 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2810596731648377 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170933512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19524312614912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617649959069696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317306040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376²–WÕ¹8N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39729230 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2791102181335040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146372528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2791102198374400 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚18449 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2825064098869248 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2805540144640000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146372530 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72048026 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32318796 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _runtime‚15667.640533913 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317316033 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32318793 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170933512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72047758 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146372528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19523971268608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617649940457472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19738232039936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2825064457254912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146372530 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2810596731648377 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170933512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19524312614912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.al!nž located‚617649959069696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317306040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2805540144640000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19738147664384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19523971268608 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2825064457254912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚637388088121856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2805540127600640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170943503 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39728965 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170943503 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19524312614912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19494533273977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088ð ·ÈøÕ4º8/ _wandbruntime‚15667  _runtime‚15667!h 7»8‚ 1loss/SRqqt2/detection_loss‚0.5865723805810921 epoch‚46 $ _timestamp‚1.784288475517897e+09 _runtime‚15667.643317095 /loss/FRt1/assignment_loss‚1.410489455802839 1loss/FBt1/assignment_loss‚0.02568079670654595 0loss/FBt1/detection_loss‚0.34484962498550936 %loss/total_loss‚6.539022737659432 trainer/global_step‚18449 _step‚1521 2loss/SRqqt2/assignment_loss‚0.3963083530067329 2loss/FBt2/assignment_loss‚0.004466995147722706 /loss/FRt1/detection_loss‚0.4803966588511456 0loss/SRqqt1/detection_loss‚0.750067260678747 /loss/FBt2/detection_loss‚0.3317894890808083 /loss/FRt2/assignment_loss‚0.816262193234034 2loss/SRqqt1/assignment_loss‚0.2877961551052846 .loss/FRt2/detection_loss‚1.104343374478971VÀ ª ¼8‡ 2loss/SRqqt1/assignment_loss‚0.2877961551052846 .loss/FRt2/detection_loss‚1.104343374478971 1loss/SRqqt2/detection_loss‚0.5865723805810921 epoch‚46 $ _timestamp‚1.784288475517897e+09 _runtime‚15667.643317095 /loss/FRt1/assignment_loss‚1.410489455802839 1loss/FBt1/assignment_loss‚0.02568079670654595 0loss/FBt1/detection_loss‚0.34484962498550936 %loss/total_loss‚6.539022737659432 trainer/global_step‚18449 _step‚1521 2loss/SRqqt2/assignment_loss‚0.3963083530067329 2loss/FBt2/assignment_loss‚0.004466995147722706 /loss/FRt1/detection_loss‚0.4803966588511456 0loss/SRqqt1/detection_loss‚0.750067260678747 /loss/FBt2/detection_loss‚0.3317894890808083 /loss/FRt2/assignment_loss‚0.816262193234034ñ úX²Ð“½8‚(:ˆ â©èÒ€Ðठgpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÄ-#%¾8‚(:š â©èÒòÊó¥ disk.rbd3.in‚0& disk.nvme0n1.out‚70904.34130859375 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625# proc.memory.rssMB‚ 2809.57421875% disk.nvme0c0n1.in‚145023.25390625 disk.drbd2165.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.148280143737793 proc.cpu.threads‚17 disk.sda3.in‚ 0.0078125( proc.memory.availableMB‚ 335.66796875$ disk./.usageGB‚550.5889320373535 disk.sda2.out‚0 disk.dm-1.out‚ 661.4296875 disk.sda1.in‚0( disk.nvme0c0n1.out‚70904.34130859375 disk.sda.out‚ 661.4296875 disk.dm-0.in‚145023.25390625$ memory_percent‚97.95124530792236 network.recv‚ 4.142568e+06 disk.sda.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.drbd2165.in‚0 disk.dm-1.in‚ 0.0078125# disk.dm-0.out‚70904.34130859375 disk.rbd0.in‚0 cpu‚26.004442440903098 network.sent‚ 2.9428008e+07* disk./.usagePercent‚36.950485203911434 disk.sda3.out‚ 661.4296875! disk.rbd1.out‚122.84423828125ÑnÛô“¿8‚(:ˆ ñ©èÒª—”¤ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚243.248' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.6496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19573995482624 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2812754955091968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚1523 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚15706.995734681 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2832328592196096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚619287949625344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2817825002413929 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171388889 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318125408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19573654143488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2812754938052608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19543798157161 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171378912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39831616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚639076468940288 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2817825360003179 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2798281204256768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2812754955091968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318135387 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19788500702720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2832328592196096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19573654143488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32404298 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2812754938052608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚639076359653888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19544138707051ËŒ  (È8…P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2832328950574592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72235658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32404301 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146746498 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2817825002413929 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171388889 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318125408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19573654143488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2812754938052608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19543798157161 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171378912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39831616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚639076468940288 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2817825360003179 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2798281204256768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2812754955091968 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318135387 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19788500702720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2832328592196096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19573654143488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32404298 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2812754938052608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚639076359653888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19544138707051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146746496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19573995482624 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2812754955091968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚1523 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚15706.995734681 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2832328592196096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚619287949625344 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚639110922412544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39833666 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2832477204238336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2812902179082240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.fr̾P#eed‚ 318142092 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146754130 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72239445 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2812902196121600 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19574666770432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32406043 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19575008116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146754130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72239712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2832476845852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2817972524139523 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318152085 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146754128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19575008116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171387964 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚619321378383872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171387964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19789430555136 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171397955 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19574666770432 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2798427714928640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32406046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39833402 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171397955 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19574666770432 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2798427714928640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32406046 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39833402 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚639110922412544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39833666 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2832477204238336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2812902179082240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318142092 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146754130 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72239445 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2812902196121600 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19574666770432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32406043 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19575008116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146754130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72239712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2832476845852672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2817972524139523 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318152085 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146754128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19575008116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171387964 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚619321378383872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171387964 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19789430555136 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚18549 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2820117006573568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2820116989534208 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2825201510083687 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚15746.749446081 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1527 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147128096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2839741697558016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19839681408512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2839741697558016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚640798984577536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19594755193959 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171833364 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72426925 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318961460 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2820116989534208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19624349645312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32491551 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2820117006573568 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147128096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19839597025792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19594414644069 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171843341 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39935374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39935114 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚640799087572480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147128098 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7842885546236553e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171833364 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2839741339179520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620959387551744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2839741339179520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32491548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2805606754889728 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2805606737850368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147128098 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620959406163968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2825201152494437 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19624690984448 _LDeviceStatsMonitor.on_train_batch_starF“‚t/allocated_bytes.small_pool.allocated‚19624690984448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968B¯È2 (Ø8†P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19624690984448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318971439 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚18549 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2820117006573568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2820116989534208 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2825201510083687 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚15746.749446081 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1527 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147128096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2839741697558016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19839681408512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2839741697558016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚640798984577536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19594755193959 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171833364 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72426925 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318961460 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2820116989534208 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19624349645312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32491551 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2820117006573568 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147128096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19839597025792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19594414644069 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171843341 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39935374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39935114 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚640799087572480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147128098 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7842885546236553e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171833364 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2839741339179520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620959387551744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2839741339179520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32491548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2805606754889728 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2805606737850368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147128098 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620959406163968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2825201152494437 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19624690984448÷ ¸õ%ª4Ù8/ _wandbruntime‚15747  _runtime‚15747¸SÅ'Ú8‹N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2839889951221760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19840705577984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147135730 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39937174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19625703618560 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2820264230563840 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318988137 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318988137 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171842416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32493296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147135730 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72430467 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39937440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171852407 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2820264247603200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚640833437512704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620992816310272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19840621202432 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19595425697791 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147135728 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171842416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19625362272256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2805753265561600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842885552348254e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚640833540500480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19625362272256 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2839889951221760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2825348674220031 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147135728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171852407 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚18549 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19625703618560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19595766247793 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169½ôš³'Û8N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147135728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171852407 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚18549 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19625703618560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19595766247793 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2839889951221760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19840705577984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147135730 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39937174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19625703618560 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2820264230563840 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318988137 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318988137 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171842416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32493296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147135730 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72430467 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39937440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171852407 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2820264247603200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚640833437512704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620992816310272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19840621202432 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19595425697791 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147135728 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171842416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19625362272256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2805753265561600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7842885552348254e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚640833540500480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19625362272256 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2839889951221760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2825348674220031ø åÄþ4Ü8/ _wandbruntime‚15747  _runtime‚15747´Gœ²Ý8Š epoch‚46 _step‚1529 0loss/FRt1/assignment_loss‚1.4126753374770447 3loss/SRqqt1/assignment_loss‚0.39819021048897085 3loss/FBt1/assignment_loss‚0.0010009107234020414 4loss/FBt2/assignment_loss‚0.00011046945664851986 0loss/FRt1/detection_loss‚0.46128266671867796 /loss/FRt2/detection_loss‚1.2992365395204912 0loss/SRqqt1/detection_loss‚0.837498232310801 trainer/global_step‚18549 /loss/FRt2/assignment_loss‚0.702505912465145 1loss/SRqqt2/detection_loss‚0.5794920203526606 0loss/FBt1/detection_loss‚0.31633531834790224 # _timestamp‚1.78428855523827e+09 2loss/SRqqt2/assignment_loss‚0.4787422435834823 _runtime‚15747.3636LX'^00478 /loss/FBt2/detection_loss‚0.4485296354721738 &loss/total_loss‚6.9355994969173995Ç’­Þ8 0loss/FRt1/assignment_loss‚1.4126753374770447 3loss/SRqqt1/assignment_loss‚0.39819021048897085 3loss/FBt1/assignment_loss‚0.0010009107234020414 4loss/FBt2/assignment_loss‚0.00011046945664851986 0loss/FRt1/detection_loss‚0.46128266671867796 /loss/FRt2/detection_loss‚1.2992365395204912 0loss/SRqqt1/detection_loss‚0.837498232310801 trainer/global_step‚18549 /loss/FRt2/assignment_loss‚0.702505912465145 1loss/SRqqt2/detection_loss‚0.5794920203526606 0loss/FBt1/detection_loss‚0.31633531834790224 # _timestamp‚1.78428855523827e+09 2loss/SRqqt2/assignment_loss‚0.4787422435834823 _runtime‚15747.363600478 /loss/FBt2/detection_loss‚0.4485296354721738 &loss/total_loss‚6.9355994969173995 epoch‚46 _step‚1529ù Öß8‚(:’ ­ªèÒ™ƒ£ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ˆ¶/à8‚(:” ­ªèÒéŠô¤ disk.rbd3.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚70959.95947265625# disk.dm-0.out‚70959.95947265625# disk.nvme0n1.in‚145023.25390625 disk.sda.out‚ 663.1171875) disk./.usagePercent‚36.95281589650316 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 663.1171875# proc.memory.rssMB‚ 2809.57421875& disk.nvme0n1.out‚70959.95947265625% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 663.1171875$ disk./.usageGB‚550.6236610412598 disk.drbd2165.out‚0# proc.memory.availableMB‚335.375 cpu‚26.003095316166732 network.recv‚ 4.162327e+06 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.148280143737793 network.sent‚ 2.9556561e+07! disk.rbd1.out‚123.09423828125 disk.rbd3.out‚0 disk.dm-0.in‚145023.25390625 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125$ memory_percent‚97.95303344726562 proc.cpu.threads‚17m ,o“á8‚(:ˆ ¼ªèÒèø»¢ gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚268.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼µ½$â8‚(:™ ¼ªèÒåºî¤ disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 663.38671875 network.recv‚ 4.166524e+06) disk./.usagePercent‚36.95319069153329 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 663.38671875# proc.memory.rssMB‚ 2809.57421875 proc.cpu.threads‚17$ disk./.usageGB‚550.6292457580566& disk.nvme0n1.out‚70971.70751953125 disk.sda1.in‚0 disk.sda.in‚ 0.0078125! disk.rbd1.out‚123.09423828125 disk.rbd3.in‚0* proc.memory.percent‚17.148280143737793( disk.nvme0c0n1.out‚70971.70751953125 disk.sda.out‚ 663.38671875 disk.dm-0.in‚145023.25390625% proc.memory.availableMB‚ 335.78125 cpu‚26.002792048870553 network.sent‚ 2.9619638e+07 disk.drbd2165.in‚0% disk.nvme0c0n1.in‚145023.25390625# disk.dm-0.out‚70971.70751953125# disk.nvme0n1.in‚145023.25390625 disk.sda2.in‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.95055389404297»èvŒžã8‚(:“ ˪èÒ̯¢ gpu.0.gpu‚85.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.314' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.062799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eñ¶B"ä8‚(:— ˪èÒ«Ò¤ network.recv‚ 4.170098e+06 disk.rbd1.in‚0.0625 disk.drbd2165.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 663.62109375 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 663.62109375 disk.sda.out‚ 663.62109375 disk.sda2.in‚0 cpu‚26.002537428594025 disk.rbd3.out‚0" disk.dm-0.out‚70981.5498046875 disk.drbd2165.out‚0( disk./.usagePercent‚36.9535908313093$ disk./.usageGB‚550.6352081298828! disk.rbd1.out‚123.21923828125 proc.cpu.threads‚17# disk.nvme0n1.in‚145023.25390625% disk.nvme0n1.out‚70981.5498046875 disk.sda.in‚ 0.0078125$ memory_percent‚97.95355796813965 network.sent‚ 2.9624726e+07 disk.rbd0.in‚0' disk.nvme0c0n1.out‚70981.5498046875 disk.dm-0.in‚145023.25390625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125' proc.memory.availableMB‚ 335.2890625# proc.memory.rssMB‚ 2809.57421875* proc.memory.percent‚17.148280143737793¢ÿèL4å8/ _wandbruntime‚15786  _runtime‚15786p¾ªÐÊæ8Ä & lr-AdamW/pg1‚5.933821735532612e-06 & lr-AdamW/pg2‚5.933821735532612e-06 trainer/global_step‚18599 % _timestamp‚1.7842885945953224e+09 _runtime‚15786.720918506 _step‚1530g0'Ïç8É trainer/global_step‚18599 % _timestamp‚1.7842885945953224e+09 _runtime‚15786.720918506 _step‚1530 & lr-AdamW/pg1‚5.933821735532612e-06 & lr-AdamW/pg2‚5.933821735532612e-06ú "Mÿ4è8/ _wandbruntime‚15786  _runtime‚15786à`NL(é8P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 319797512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2847154086162944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19675045147136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19890872959488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19675386486272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2847154086162944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319807491 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72618548 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2827479058055168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172297793 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40040013 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32578798 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19645371680867 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2847154444541440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2832577302574945 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147509698 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚15786.722167043 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19675386486272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7842885945963886e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172287816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2847154444541440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚642521801432576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147509696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40039750 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622630844090368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172297793 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147509696 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2832577660164195 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2812932288483328 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚642521698437632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622630825478144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19645031130977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2847154086162944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19675045147136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19890872959488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19675386486272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2847154086162944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319807491 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72618548 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2827479058055168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172297793 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40040013 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32578798 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19645371680867 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2847154444541440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2832577302574945 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147509698 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚15786.722167043 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19675386486272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7842885945963886e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172287816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2847154444541440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚642521801432576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147509696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40039750 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622630844090368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172297793 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147509696 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2832577660164195 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2812932288483328 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚642521698437632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622630825478144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19645031130977 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319824189 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚642556139043840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚622664254236672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2827626299084800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚622664272848896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172306859 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32580543 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642556250420224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172296868 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147517330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147517330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32580546 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2847302698205184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319814196 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19676399120384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40041845 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784288595207188e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2827626299084800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19676057774080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19676057774080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19891884807168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19676399120384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceSD4 ntatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2827626282045440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72622388 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2813078782115840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72622653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2832725181889901 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚18599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2847302339819520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147517328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19891977571328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1532 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2847302698205184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172296868 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19646042184699 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19646382734701 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147517328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319814196 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40042107 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172296868 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147517330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147517330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32580546 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2847302698205184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319814196 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19676399120384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40041845 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784288595207188e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2827626299084800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19676057774080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19676057774080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19891884807168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19676399120384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2827626282045440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72622388 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2813078782115840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72622653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2832725181889901 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚18599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2847302339819520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147517328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19891977571328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1532 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2847302698205184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172296868 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19646042184699 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19646382734701 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147517328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319814196 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40042107 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319824189 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚642556139043840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚622664254236672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2827626299084800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚622664272848896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172306859 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32580543 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642556250420224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FQó8‚(:’ éªèÒùÝ¢ gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´¤žfô8‚(:‘ éªèÒ­Óê¤ disk.rbd0.out‚0* proc.memory.percent‚17.148280143737793# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚145023.25390625! disk.rbd1.out‚123.21923828125 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 663.98046875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0' disk./.usagePercent‚36.954369862236 disk.dm-0.in‚145023.25390625 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚71002.34765625 disk.sda2.out‚0 network.sent‚ 2.9692895e+07 network.recv‚ 4.177893e+06 disk.dm-1.out‚ 663.98046875 disk.rbd1.in‚0.0625( proc.memory.availableMB‚ 335.22265625# disk.nvme0n1.out‚71002.34765625 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚97.95396327972412 cpu‚26.002011623522538 proc.cpu.threads‚17$ disk./.usageGB‚550.6468162536621 disk.dm-0.out‚71002.34765625 disk.sda1.out‚0 disk.sda.out‚ 663.98046875 disk.dm-1.in‚ 0.0078125# proc.memory.rssMB‚ 2809.57421875îôatžõ8‚(:“ øªèÒ«é¢ gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å3Þ#ö8‚(:˜ øªèÒÌÈ–¥ network.sent‚ 2.9698048e+07 disk.rbd3.out‚0# disk.dm-0.out‚71017.02294921875 proc.cpu.threads‚17# disk.nvme0n1.in‚145023.25390625( disk.nvme0c0n1.out‚71017.02294921875% disk.nvme0c0n1.in‚145023.25390625$ disk./.usageGB‚550.6534805297852 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.dm-0.in‚145023.25390625) disk./.usagePercent‚36.95481710739831& disk.nvme0n1.out‚71017.02294921875 disk.sda.in‚ 0.0078125 disk.dm-1.out‚ 664.24609375* proc.memory.percent‚17.148280143737793 cpu‚26.00170924772485 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0$ memory_percent‚97.95398712158203% proc.memory.availableMB‚ 335.21875 network.recv‚ 4.181149e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125# proc.memory.rssMB‚ 2809.57421875 disk.sda.out‚ 664.24609375 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 664.24609375! disk.rbd1.out‚123.28173828125„°4÷8/ _wandbruntime‚15826  _runtime‚15826ϸ jÈø8 % lr-AdamW/pg1‚5.51946479004488e-06 % lr-AdamW/pg2‚5.51946479004488e-06 trainer/global_step‚18649 % _timestamp‚1.7842886345447261e+09 _runtime‚15826.670355458 _step‚1534f}Ì6Íù8Ç % lr-AdamW/pg2‚5.51946479004488e-06 trainer/global_step‚18649 % _timestamp‚1.7842886345447261e+09 _runtime‚15826.670355458 _step‚1534 % lr-AdamW/pg1‚5.51946479004488e-06þ 6Õµ4ú8/ _wandbruntime‚15826  _runtime‚15826†,X=(û8P H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19695988167775 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172742268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2854566833146368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2820257805037568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172752245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2854567191524864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19725740648960 trainer/global_step‚18649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2834841092497408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320643543 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19942201574400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644244572168192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644244464978944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72810481 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32666048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624302282016768 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2839953452655453 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2834841092497408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19726081988096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624302263404544 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147891296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172742268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842886345457747e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147891296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2820257822076928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19725740648960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19726081988096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚15826.674372561 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147891298 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2854567191524864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2834841109536768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32666051 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320633564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2854566833146368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40144433 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2834841109536768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40144698 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172752245 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72810749 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 J+•wü8†P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚1535 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19695647617885 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19695988167775 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172742268 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2854566833146368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2820257805037568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172752245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2854567191524864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19725740648960 trainer/global_step‚18649 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190¬µ¶6•607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2834841092497408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320643543 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19942201574400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644244572168192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644244464978944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72810481 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32666048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624302282016768 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2839953452655453 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2834841092497408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19726081988096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624302263404544 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147891296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172742268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7842886345457747e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147891296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2820257822076928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19725740648960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19726081988096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _runtime‚15826.674372561 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147891298 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2854567191524864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2834841109536768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32666051 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320633564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2854566833146368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40144433 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2834841109536768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40144698 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172752245 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72810749 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2834988350566400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172761311 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644279020893696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2840100974381047 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19696999221609 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚15827.282714809 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19726753275904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320650248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚18649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320660241 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147898928 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2834988333527040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72814542 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624335692163072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320660241 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172751320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842886351569102e+09 _step‚1536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320650248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2834988333527040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644278909517312 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2834988350566400 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2854715445188608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19726753275904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172751320 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72814258 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2840101331970409 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2820404332748800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32667793 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2820404315709440 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147898930 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19727094622208 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320660241 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172751320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7842886351569102e+09 _step‚1536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320650248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2834988333527040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚644278909517312 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2834988350566400 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2854715445188608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19726753275904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172751320 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72814258 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2840101331970409 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2820404332748800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32667793 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2820404315709440 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147898930 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19727094622208 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2834988350566400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172761311 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚644279020893696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2840100974381047 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19696999221609 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚15827.282714809 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19726753275904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320650248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚18649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320660241 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147898928 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2834988333527040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72814542 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624335692163072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2861979580129792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321469616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2861979580129792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚645967298822144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2842203161018368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚625973701330944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2842203161018368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19776436150784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73003192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40249620 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173206697 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2861979938508288 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19776777489920 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148272896 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2827583355670528 _runtime‚15866.400613486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚18699 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148272898 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19776777489920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2827583338631168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19993494496256 _step‚1539 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40249891 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚625973719943168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32753298ßx—€ (Œ9†P VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2847329602735961 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19746264104793 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2861979580129792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321469616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2861979580129792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚645967298822144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2842203161018368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚625973701330944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2842203161018368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19776436150784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73003192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40249620 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173206697 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2861979938508288 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19776777489920 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148272896 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2827583355670528 _runtime‚15866.400613486 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚18699 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148272898 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19776777489920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2827583338631168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19993494496256 _step‚1539 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40249891 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚625973719943168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32753298 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73002918 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2861979938508288 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19993578878976 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2847329960325211 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148272898 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32753301 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2842203143979008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7842886742748291e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148272896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173206697 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2842203143979008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321479595 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173196720 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173196720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚645967195827200 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2842350402048000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73007043 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2847477124461555 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173215763 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19994606216704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2842350402048000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19777448777728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148280530 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321496293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626007148701696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2842350385008640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40251747 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚15867.012653333 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2827729866342400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2862127833786368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19747275158515 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626007130089472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148280530 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148280528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32755046 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2827729849303040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148280528 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321496293 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173205772 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2862128192172032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2842350385008640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1540 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73006790 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40251997 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19994528132608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842886748866649e+09æ¬0Z9N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚646001658222080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32755043 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173205772 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19777448777728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173215763 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19994606216704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2842350402048000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19777448777728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148280530 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321496293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626007148701696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2842350385008640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40251747 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚15867.012653333 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2827729866342400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2862127833786368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19747275158515 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626007130089472 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148280530 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148280528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32755046 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2827729849303040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148280528 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321496293 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173205772 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2862128192172032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2842350385008640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1540 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73006790 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40251997 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19994528132608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842886748866649e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2842350402048000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73007043 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2847477124461555 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19827131652608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚647690034891776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148654496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73194493 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20044877022208 _step‚1543 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2849565212499968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚15906.368756347 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2869392327113216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2869392327113216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2834908889264128 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19827131652608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2869392685491712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173661149 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32840551 trainer/global_step‚18749 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322315647 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2854705752816469 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19796880591701 % _timestamp‚1.7842887142429743e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979â—onGž9†P YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173661149 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2834908872224768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2849565212499968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚15906.368756347 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2869392327113216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2869392327113216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2834908889264128 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19827131652608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2869392685491712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173661149 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32840551 trainer/global_step‚18749 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322315647 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2854705752816469 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19796880591701 % _timestamp‚1.7842887142429743e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2854706110405719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M?DeviceS¤ú,Å tatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173651172 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2869392685491712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19827472991744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚627645157869568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 322305668 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73194761 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19827131652608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚647690034891776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148654496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73194493 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20044877022208 _step‚1543 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _runtime‚15906.979320724 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19828485625856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2835055399936000 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚18749 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73198305 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19828144279552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322322352 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19798232195425 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19828144279552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173670215 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73198589 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚627678586628096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚1544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2849712453529600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40356293 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2869540939155456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322332345 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173660224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2835055382896640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ‹P{'¡9N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19828485625856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2835055399936000 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚18749 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73198305 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19828144279552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322322352 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19798232195425 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19828144279552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173670215 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73198589 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚627678586628096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _step‚1544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2849712453529600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40356293 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2869540939155456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322332345 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173660224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2835055382896640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2849712453529600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32842293 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚627678568015872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148662128 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2849712436490240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32842296 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _runtime‚15906.979320724 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281ˆ ÛU4¢9/ _wandbruntime‚15906  _runtime‚15906g ù £9‡ 1loss/FBt1/assignment_loss‚0.00427120785098037 %loss/total_loss‚7.103603428230411 _step‚1545 2loss/SRqqt1/assignment_loss‚0.3286056109388333 2loss/SRqqt2/assignment_loss‚0.4451509404629655 1loss/SRqqt2/detection_loss‚0.5876375969104151 /loss/FBt2/detection_loss‚0.5088388092060059 epoch‚46 0loss/FRt2/assignment_loss‚0.8083448752843825 4loss/FBt2/assignment_loss‚0.00011136151996433428 /loss/FRt1/detection_loss‚0.4926190817818921 /loss/FRt2/detection_loss‚1.1576474854742436 _runtime‚15906.982272747 0loss/FRt1/assignment_loss‚1.4998399509809404 1loss/SRqqt1/detection_loss‚0.7361297032290555 /loss/FBt1/detection_loss‚0.5344068045907321 trainer/global_step‚18749 $ _timestamp‚1.784288714856913e+09ÂT¤9Œ epoch‚46 0loss/FRt2/assignment_loss‚0.8083448752843825 4loss/FBt2/assignment_loss‚0.00011136151996433428 /loss/FRt1/detection_loss‚0.4926190817818921 /loss/FRt2/detection_loss‚1.1576474854742436 _runtime‚15906.982272747 0loss/FRt1/assignment_loss‚1.4998399509809404 1loss/SRqqt1/detection_loss‚0.7361297032290555 /loss/FBt1/detection_loss‚0.5344068045907321 trainer/global_step‚18749 $ _timestamp‚1.784288714856913e+09 1loss/FBt1/assignment_loss‚0.00427120785098037 %loss/total_loss‚7.103603428230411 _step‚1545 2loss/SRqqt1/assignment_loss‚0.3286056109388333 2loss/SRqqt2/assignment_loss‚0.4451509404629655 1loss/SRqqt2/detection_loss‚0.5876375969104151 /loss/FBt2/detection_loss‚0.5088388092060059‰ ÿ‰ä“¥9‚(:ˆ Ò«èÒå‚Ó¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MS_~¦9‚(: Ò«èÒ¶ýé¤$ disk./.usageGB‚550.6772499084473 disk.rbd1.in‚0.0625 disk.dm-0.in‚145023.25390625 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda.out‚ 665.90625 proc.cpu.threads‚17" disk.dm-0.out‚71083.9130859375 disk.sda3.out‚ 665.90625# disk.nvme0n1.in‚145023.25390625' disk.nvme0c0n1.out‚71083.9130859375! disk.rbd1.out‚123.46923828125% disk.nvme0n1.out‚71083.9130859375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚145023.25390625* proc.memory.percent‚17.148280143737793 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0 disk.sda1.out‚0 network.recv‚ 4.202226e+06) disk./.usagePercent‚36.95641229034416 disk.dm-1.in‚ 0.0078125 network.sent‚ 2.9843675e+07 disk.dm-1.out‚ 665.90625 disk.sda2.in‚0( proc.memory.availableMB‚ 335.23046875# proc.memory.rssMB‚ 2809.57421875 cpu‚26.000103107640395 disk.sda.in‚ 0.0078125 disk.sda2.out‚0# memory_percent‚97.9539155960083ÚI¯“§9‚(:ˆ á«èÒ×áÅ¢ gpu.0.gpu‚88.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D|&¨9‚(:› á«èÒ½àý¤$ disk./.usageGB‚550.6871376037598 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125$ memory_percent‚97.95255661010742& proc.memory.availableMB‚ 335.453125( disk.nvme0c0n1.out‚71092.79833984375# proc.memory.rssMB‚ 2809.57421875 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda.out‚ 666.12890625 disk.sda3.out‚ 666.12890625! disk.rbd1.out‚123.46923828125 cpu‚25.999836459006396 proc.cpu.threads‚17* disk./.usagePercent‚36.957075861872916 disk.drbd2165.in‚0 disk.dm-1.out‚ 666.12890625 network.sent‚ 2.9906759e+07 disk.dm-0.in‚145023.25390625# disk.nvme0n1.in‚145023.25390625 disk.sda3.in‚ 0.0078125* proc.memory.percent‚17.148280143737793 disk.rbd3.in‚0# disk.dm-0.out‚71092.79833984375 disk.rbd0.out‚0& disk.nvme0n1.out‚71092.79833984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.rbd1.in‚0.0625 network.recv‚ 4.206966e+06þî]D‘©9‚(:† ð«èÒå²ä£ gpu.0.gpu‚37.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶%QŸ%ª9‚(:š ð«èÒÐ̤¥( disk.nvme0c0n1.out‚71105.87353515625) disk./.usagePercent‚36.95747267355099! disk.rbd1.out‚123.53173828125 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚550.6930503845215# disk.dm-0.out‚71105.87353515625& proc.memory.availableMB‚ 335.203125 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 666.48828125# disk.nvme0n1.in‚145023.25390625% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 4.210207e+06 disk.rbd1.in‚0.0625 disk.sda1.in‚0# proc.memory.rssMB‚ 2809.57421875& disk.nvme0n1.out‚71105.87353515625 disk.drbd2165.out‚0 disk.sda.out‚ 666.48828125 disk.sda2.out‚0$ memory_percent‚97.95408248901367 proc.cpu.threads‚17 disk.dm-1.out‚ 666.48828125 disk.rbd0.out‚0 cpu‚25.999585426940747 network.sent‚ 2.9911896e+07 disk.dm-0.in‚145023.25390625* proc.memory.percent‚17.148280143737793‘§ä4«9/ _wandbruntime‚15946  _runtime‚15946-LwUɬ9à $ _timestamp‚1.784288754209073e+09 _runtime‚15946.334685233 _step‚1546 & lr-AdamW/pg1‚4.365444403505701e-06 & lr-AdamW/pg2‚4.365444403505701e-06 trainer/global_step‚18799;†8÷έ9È trainer/global_step‚18799 $ _timestamp‚1.784288754209073e+09 _runtime‚15946.334685233 _step‚1546 & lr-AdamW/pg1‚4.365444403505701e-06 & lr-AdamW/pg2‚4.365444403505701e-06Š Ì–ÃS4®9/ _wandbruntime‚15946  _runtime‚15946pjABö ¯9€P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19878168493568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚649412656877056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174115601 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73386076 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2876805432475136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚649412755677696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19877827154432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40458278 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19877827154432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629316577183744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40458519 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32927798 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174115601 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 _step‚1547 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2876805074096640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19847497078609 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842887542101393e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174105624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2856927263981568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32927801 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2876805074096640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323141720 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2856927246942208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149036096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149036096 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2876805432475136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20096159881728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2856927246942208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚18799 _runtime‚15946.335906909 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149036098 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19878168493568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2862082260486227 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888C÷Ç (°9…P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629316595795968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20096079693312 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2876805074096640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19847497078609 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7842887542101393e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174105624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2856927263981568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32927801 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2876805074096640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323141720 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2856927246942208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149036096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149036096 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2876805432475136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20096159881728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2856927246942208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚18799 _runtime‚15946.335906909 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149036098 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19878168493568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2862082260486227 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19878168493568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚649412656877056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174115601 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73386076 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2876805432475136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚649412755677696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19877827154432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40458278 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19877827154432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629316577183744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40458519 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32927798 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174115601 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 _step‚1547 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2876953327753216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2857074505011200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629350005942272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20097188155392 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19848508132331 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2857074487971840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323158404 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2862229782211933 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19879181127680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149043728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19878839781376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1548 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174114676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19878839781376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149043730 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73389857 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2876953686138880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323168397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚649447099236352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32929543 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2842380933529600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323158404 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚18799 _runtime‚15946.947461815 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20097093294080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0m“R}³9N QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2876953327753216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2857074505011200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629350005942272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20097188155392 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19848508132331 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2857074487971840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323158404 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2862229782211933 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19879181127680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149043728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19878839781376 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚QÂï®™1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚1548 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174114676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19878839781376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149043730 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73389857 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2876953686138880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323168397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚649447099236352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32929543 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2842380933529600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323158404 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 trainer/global_step‚18799 _runtime‚15946.947461815 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20097093294080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Œ õyî4´9/ _wandbruntime‚15946  _runtime‚15946ž›ª.µ9Š 1loss/SRqqt2/detection_loss‚0.6011105969065846 /loss/FBt2/detection_loss‚0.6001946345624876 epoch‚46 % _timestamp‚1.7842887548248003e+09 /loss/FBt1/detection_loss‚0.3554051250843049 %loss/total_loss‚7.039151711692653 2loss/SRqqt1/assignment_loss‚0.3155238984348385 2loss/SRqqt2/assignment_loss‚0.4306197303085137 trainer/global_step‚18799 0loss/FRt1/assignment_loss‚1.3644012544333204 0loss/FRt2/assignment_loss‚0.8287758006082548 2loss/FBt2/assignment_loss‚0.002132378330903632 _runtime‚15946.950136104 _step‚1549 4loss/FBt1/assignment_loss‚0.00012931105354541117 0loss/FRt1/detection_loss‚0.49952039541402066 /loss/FRt2/detection_loss‚1.1735254948574045 1loss/SRqqt1/detection_loss‚0.8678130916984736~°“å¶9 _step‚1549 4loss/FBt1/assignment_loss‚0.00012931105354541117 0loss/FRt1/detection_loss‚0.49952039541402066 /loss/FRt2/detection_loss‚1.1735254948574045 1loss/SRqqt1/detection_loss‚0.8678130916984736 1loss/SRqqt2/detection_loss‚0.6011105969065846 /loss/FBt2/detection_loss‚0.6001946345624876 epoch‚46 % _timestamp‚1.7842887548248003e+09 /loss/FBt1/detection_loss‚0.3554051250843049 %loss/total_loss‚7.039151711692653 2loss/SRqqt1/assignment_loss‚0.3155238984348385 2loss/SRqqt2/assignment_loss‚0.4306197303085137 trainer/global_step‚18799 0loss/FRt1/assignment_loss‚1.3644012544333204 0loss/FRt2/assignment_loss‚0.8287758006082548 2loss/FBt2/assignment_loss‚0.002132378330903632 _runtime‚15946.950136104 9Uh‘·9‚(:† ÿ«èÒÕ욢 gpu.0.gpu‚43.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0² :9¸9‚(:Œ ÿ«èÒߎ¸¥ proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.sda.out‚666.875 disk.sda3.out‚666.875$ memory_percent‚97.96242713928223# disk.nvme0n1.in‚145023.25390625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚666.875 disk.dm-0.in‚145023.25390625) disk./.usagePercent‚36.95836741988316 disk.rbd0.out‚0( disk.nvme0c0n1.out‚71118.05029296875 disk.dm-1.in‚ 0.0078125 disk.sda1.in‚0% disk.nvme0c0n1.in‚145023.25390625 network.sent‚ 2.9916944e+07 network.recv‚ 4.213737e+06# proc.memory.rssMB‚ 2810.88671875* proc.memory.percent‚17.156291007995605 cpu‚26.002506208005684 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0! disk.rbd1.out‚123.53173828125 disk.rbd1.in‚0.0625# disk.dm-0.out‚71118.05029296875$ disk./.usageGB‚550.7063827514648& disk.nvme0n1.out‚71118.05029296875 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125' proc.memory.availableMB‚ 333.8359375^Œ°K4¹9/ _wandbruntime‚15964  _runtime‚15964ŸS§ †=º9€{ @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7688742876052856 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875089526176453 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25073671340942383 +particle/specificity‚0.9036664366722107 6validation_average_jet_accuracy‚0.3715430796146393 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26420795917510986 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7760167121887207 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6550041437149048 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5208153128623962 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6302253007888794 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8022332191467285 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2692008316516876 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48734766244888306 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 )jet/accuracy_1_of_3‚0.923831045627594 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2180105596780777 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26377642154693604 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _step‚1550 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6255806684494019 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6238884329795837 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037037014961243 *jet/accuracy_2_of_4‚0.8028580546379089 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6240143179893494 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 trainer/global_step‚18799 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897313714027405 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2101117968559265 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6089917421340942 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2634865343570709 *jet/accuracy_2_of_3‚0.5739680528640747 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3554040491580963 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5302414298057556 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04592849686741829 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8748446702957153 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6554311513900757 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29607853293418884 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5724626183509827 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022613270208239555 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 # _timestamp‚1.78428877188954e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9134987592697144 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947301149368286 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚8.086075558141268 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02115723118185997 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6232848763465881 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5927108526229858 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6598905324935913 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7959400415420532 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_3‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0401957705616951 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6933505535125732 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6000219583511353 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6582722663879395 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9526143074035645 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.586990237236023 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6199004650115967 (particle/accuracy‚0.8416961431503296 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24922877550125122 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7335600852966309 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.236668199300766 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20732314884662628 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32087963819503784 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3551636338233948 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0436660572886467 *jet/accuracy_1_of_1‚0.3279196619987488 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6422677040100098 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8452844023704529 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5316729545593262 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23987869918346405 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2734634280204773 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_4‚0.9830800294876099 +jet/accuracy_3_of_4‚0.07847467809915543 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27487850189208984 *jet/accuracy_1_of_2‚0.7882296442985535 +jet/accuracy_3_of_3‚0.05688074603676796 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9906184673309326 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7402494549751282 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6899368166923523 jet/accuracy_3_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8748446702957153 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33006006479263306 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7217947840690613 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2429828643798828 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2848012447357178 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25547000765800476 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5801788568496704 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.15816271305084229 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2032051533460617 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2503798305988312 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.728343665599823 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5984484553337097 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22474263608455658 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27366822957992554 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037037014961243 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6331349611282349 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8976248502731323 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6605631113052368 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022459974512457848 epoch‚46 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.615737795829773 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.075925812125206 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021032636985182762 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.569647490978241 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6000219583511353 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6239431500434875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7194409966468811 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2634865343570709 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7603293657302856 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08706823736429214 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5133333206176758 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5937493443489075 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6538041234016418 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5939305424690247 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03586982190608978 _runtime‚15964.015828803 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2384580820798874 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19884295761585236 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7636415362358093 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953824281692505 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23803475499153137 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8526139855384827 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3712509274482727 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7219009399414062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6591065526008606 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_3_of_4‚1 'particle/f_score‚0.6252862215042114 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.222817063331604 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.6151062846183777 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 particle/accuracy_5_of_5‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5457562208175659 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4520525634288788 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7296673655509949 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2156539410352707 jet/accuracy_2_of_5‚1óÃŒm$»9…{ :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48734766244888306 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 )jet/accuracy_1_of_3‚0.923831045627594 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2180105596780777 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26377642154693604 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _step‚1550 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6255806684494019 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6238884329795837 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037037014961243 *jet/accuracy_2_of_4‚0.8028580546379089 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6240143179893494 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 trainer/global_step‚18799 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897313714027405 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2101117968559265 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6089917421340942 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2634865343570709 *jet/accuracy_2_of_3‚0.5739680528640747 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3554040491580963 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5302414298057556 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04592849686741829 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8748446702957153 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6554311513900757 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29607853293418884 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5724626183509827 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022613270208239555 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 # _timestamp‚1.78428877188954e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9134987592697144 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947301149368286 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚8.086075558141268 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02115723118185997 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6232848763465881 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5927108526229858 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6598905324935913 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7959400415420532 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_3‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0401957705616951 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6933505535125732 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6000219583511353 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6582722663879395 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9526143074035645 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.586990237236023 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6199004650115967 (particle/accuracy‚0.8416961431503296 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24922877550125122 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7335600852966309 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_2_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.236668199300766 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20732314884662628 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32087963819503784 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3551636338233948 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0436660572886467 *jet/accuracy_1_of_1‚0.3279196619987488 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6422677040100098 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8452844023704529 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5316729545593262 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23987869918346405 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2734634280204773 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_4‚0.9830800294876099 +jet/accuracy_3_of_4‚0.07847467809915543 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27487850189208984 *jet/accuracy_1_of_2‚0.7882296442985535 +jet/accuracy_3_of_3‚0.05688074603676796 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9906184673309326 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7402494549751282 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6899368166923523 jet/accuracy_3_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8748446702957153 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33006006479263306 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7217947840690613 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2429828643798828 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2848012447357178 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25547000765800476 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5801788568496704 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.15816271305084229 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2032051533460617 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2503798305988312 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.728343665599823 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5984484553337097 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22474263608455658 8 af;¼Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27366822957992554 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037037014961243 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6331349611282349 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8976248502731323 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6605631113052368 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022459974512457848 epoch‚46 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.615737795829773 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.075925812125206 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021032636985182762 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.569647490978241 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6000219583511353 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6239431500434875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7194409966468811 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2634865343570709 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7603293657302856 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08706823736429214 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5133333206176758 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5937493443489075 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6538041234016418 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5939305424690247 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03586982190608978 _runtime‚15964.015828803 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2384580820798874 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19884295761585236 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7636415362358093 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953824281692505 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23803475499153137 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8526139855384827 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3712509274482727 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7219009399414062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6591065526008606 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_3_of_4‚1 'particle/f_score‚0.6252862215042114 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.222817063331604 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.6151062846183777 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 particle/accuracy_5_of_5‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5457562208175659 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4520525634288788 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7296673655509949 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2156539410352707 jet/accuracy_2_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7688742876052856 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875089526176453 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25073671340942383 +particle/specificity‚0.9036664366722107 6validation_average_jet_accuracy‚0.3715430796146393 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26420795917510986 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7760167121887207 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6550041437149048 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5208153128623962 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6302253007888794 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8022332191467285 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2692008316516876 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1Ž ܇¿~“¼9‚(:ˆ Ž¬èÒøáÖ¢ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚190.423' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.0846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•³+ä ½9‚(:þ Ž¬èÒôŠè¤ disk.dm-0.in‚145023.25390625$ memory_percent‚97.96524047851562 disk.rbd3.out‚0 disk.dm-1.in‚ 0.0078125# disk.nvme0n1.in‚145023.25390625 disk.sda2.out‚0 cpu‚26.003286197012716 disk.sda.out‚ 667.28125 network.recv‚ 4.2178e+06# disk./.usageGB‚550.712646484375 disk.dm-1.out‚ 667.28125! disk.rbd1.out‚123.59423828125 disk.rbd3.in‚0 disk.dm-0.out‚71130.95703125 disk.rbd0.in‚0% disk.nvme0c0n1.out‚71130.95703125 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.sda3.out‚ 667.28125 network.sent‚ 2.9980073e+07# disk.nvme0n1.out‚71130.95703125 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0.0625 disk.sda1.in‚0% disk.nvme0c0n1.in‚145023.25390625# proc.memory.availableMB‚333.375 proc.memory.rssMB‚ 2811.03125 proc.cpu.threads‚17 disk.sda3.in‚ 0.0078125) disk./.usagePercent‚36.95878778425438 disk.sda1.out‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.15717315673828ä¿u“¾9‚(:ˆ ¬èÒÍÅ¢ gpu.0.gpu‚51.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”®b¹!¿9‚(:– ¬èÒÓì¤ disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚71141.22412109375 disk.sda.in‚ 0.0078125( proc.memory.availableMB‚ 333.42578125 cpu‚26.003020800586704 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625 disk.sda1.out‚0# memory_percent‚97.9649305343628 proc.memory.rssMB‚ 2811.03125 network.recv‚ 4.222204e+06 disk.dm-0.in‚145023.25390625 disk.sda3.out‚ 667.46484375 disk.dm-1.out‚ 667.46484375 disk.rbd0.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125) proc.memory.percent‚17.15717315673828 network.sent‚ 3.003737e+07! disk.rbd1.out‚123.59423828125 disk.rbd1.in‚0.0625 disk.rbd3.in‚0# disk.dm-0.out‚71141.22412109375 disk.sda.out‚ 667.46484375 disk.dm-1.in‚ 0.0078125$ disk./.usageGB‚550.7178421020508& disk.nvme0n1.out‚71141.22412109375 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.95913646651602·ƒ˜4À9/ _wandbruntime‚16003  _runtime‚16003¾†T`ÌÁ9Æ ' lr-AdamW/pg2‚4.0105367810025075e-06 trainer/global_step‚18849 % _timestamp‚1.7842888110646038e+09 _runtime‚16003.190161286 _step‚1551 ' lr-AdamW/pg1‚4.0105367810025075e-06õDgÑÂ9Ë ' lr-AdamW/pg1‚4.0105367810025075e-06 ' lr-AdamW/pg2‚4.0105367810025075e-06 trainer/global_step‚18849 % _timestamp‚1.7842888110646038e+09 _runtime‚16003.190161286 _step‚1551 C‘ª‡4Ã9/ _wandbruntime‚16003  _runtime‚16003( dÙ(Ä9P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174640375 _step‚1552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324178739 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149538364 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174630398 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2886165120935936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2886165479314432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33017571 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174640375 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2866218882113536 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19946255861760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20165673388544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324168760 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174630398 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2851468602626048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2851468619665408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2886165120935936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19946597200896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149538362 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2886165479314432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19916173840636 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174640375 _step‚1552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324178739 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149538364 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174630398 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2886165120935936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2886165479314432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33017571 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174640375 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2866218882113536 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19946255861760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20165673388544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324168760 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174630398 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2851468602626048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2851468619665408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2886165120935936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19946597200896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149538362 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2886165479314432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19916173840636 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚651203963039744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19947268488704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20166607071232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚651204068124672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149545994 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19916844344468 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324195437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73613687 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631037355968512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20166693543936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19917184894470 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚1553 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33019313 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324195437 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174649441 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2886313374592512 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2866366123143168ÿ©+~'È9N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174639450 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2866366106103808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324195437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73613687 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631037355968512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20166693543936 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19917184894470 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _step‚1553 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33019313 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324195437 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174649441 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2886313374592512 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2866366123143168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚651203963039744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19947268488704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20166607071232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚651204068124672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149545994 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19916844344468 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1556 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175094827 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175094827 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2893577867919360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2873580933595136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149919962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149919962 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33104818 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 _runtime‚16043.158358619 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632675383748608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2878760943586552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2873580933595136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73801310 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2873580916555776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73801568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149919964 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2893578226297856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40696747 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20217096961024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19966449777654 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20217096961024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19966449777654 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1556 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175094827 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175094827 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2893577867919360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2873580933595136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149919962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149919962 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33104818 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 _runtime‚16043.158358619 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632675383748608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360” •WãÈ4Ø9/ _wandbruntime‚16043  _runtime‚16043x¾þ 'Ù9ˆN VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33106563 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40698575 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149927596 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325031489 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149927594 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40698837 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652926936386048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19967460831376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149927596 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20218123878912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚16043.767623739 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2873728157585408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2878908465312258 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2893726479961600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175093902 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20218041600512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325021496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚1557 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2873728174624768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2858940646891520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2893726121575936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2878908107722896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325021496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175093902 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73805138 " _timestamp‚1.7842888516418e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325031489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19998305336832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19997963990528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632708793894912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2858940663930880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632708812507136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2873728174624768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2893726479961600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175103893 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚652926835495424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Š€ú»'Ú9N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2878908107722896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325021496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175093902 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73805138 " _timestamp‚1.7842888516418e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325031489 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19998305336832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19997963990528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632708793894912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2858940663930880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632708812507136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2873728174624768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2893726479961600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175103893 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚652926835495424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33106563 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40698575 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149927596 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325031489 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149927594 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40698837 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652926936386048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19967460831376 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149927596 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20218123878912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚16043.767623739 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2873728157585408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2878908465312258 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2893726479961600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175093902 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20218041600512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325021496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚1557 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2873728174624768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2858940646891520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2893726121575936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0• aEÑM4Û9/ _wandbruntime‚16043  _runtime‚16043ãªÜ9Š 3loss/SRqqt2/assignment_loss‚0.44562035182119153 1loss/FBt1/assignment_loss‚0.00296453030144076 epoch‚47 0loss/FRt2/assignment_loss‚0.7732178077978247 3loss/SRqqt1/assignment_loss‚0.35226484914397765 3loss/FBt2/assignment_loss‚3.326212821873144e-05 0loss/FRt1/detection_loss‚0.48277839750694385 1loss/SRqqt2/detection_loss‚0.5870331157247709 0loss/FBt1/detection_loss‚0.45065546975767673 0loss/FBt2/detection_loss‚0.40235135740658734 _step‚1558 0loss/FRt1/assignment_loss‚1.3¿dNü620728547346659 /loss/FRt2/detection_loss‚1.2515600517531258 0loss/SRqqt1/detection_loss‚0.823376357132882 %loss/total_loss‚6.933928405209306 trainer/global_step‚18899 $ _timestamp‚1.784288851644214e+09 _runtime‚16043.769555154tĤÝ9 /loss/FRt2/detection_loss‚1.2515600517531258 0loss/SRqqt1/detection_loss‚0.823376357132882 %loss/total_loss‚6.933928405209306 trainer/global_step‚18899 $ _timestamp‚1.784288851644214e+09 _runtime‚16043.769555154 3loss/SRqqt2/assignment_loss‚0.44562035182119153 1loss/FBt1/assignment_loss‚0.00296453030144076 epoch‚47 0loss/FRt2/assignment_loss‚0.7732178077978247 3loss/SRqqt1/assignment_loss‚0.35226484914397765 3loss/FBt2/assignment_loss‚3.326212821873144e-05 0loss/FRt1/detection_loss‚0.48277839750694385 1loss/SRqqt2/detection_loss‚0.5870331157247709 0loss/FBt1/detection_loss‚0.45065546975767673 0loss/FBt2/detection_loss‚0.40235135740658734 _step‚1558 0loss/FRt1/assignment_loss‚1.3620728547346659– Ño“Þ9‚(:ˆ Ù¬èÒ·•é¢ gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚245.138' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.0276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G°rQ ß9‚(:• Ù¬èÒñ©Ø¤ disk.rbd1.in‚0.0625 proc.memory.rssMB‚ 2811.03125 disk.drbd2165.out‚0$ disk./.usageGB‚550.7449684143066 disk.sda2.in‚0 disk.sda3.out‚ 669.08203125! disk.rbd1.out‚123.71923828125 disk.dm-0.in‚145023.25390625& disk.nvme0n1.out‚71186.56298828125 cpu‚26.00196659095333 network.sent‚ 3.0116058e+07* disk./.usagePercent‚36.960956936091996 disk.sda.in‚ 0.0078125 disk.drbd2165.in‚0% proc.memory.availableMB‚ 333.40625% disk.nvme0c0n1.in‚145023.25390625 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 disk.rbd3.out‚0# disk.dm-0.out‚71186.56298828125 disk.sda1.out‚0 network.recv‚ 4.236776e+06 disk.rbd0.out‚0( disk.nvme0c0n1.out‚71186.56298828125$ memory_percent‚97.96504974365234 disk.rbd0.in‚0 disk.sda.out‚ 669.08203125# disk.nvme0n1.in‚145023.25390625 proc.cpu.threads‚17 disk.dm-1.out‚ 669.08203125) proc.memory.percent‚17.15717315673828˜påM’à9‚(:‡ è¬èÒÝ×Æ¢ gpu.0.gpu‚23.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷¯—¦á9‚(:” è¬è҇풥$ memory_percent‚97.96504974365234) disk./.usagePercent‚36.96135477180021 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚123.78173828125& disk.nvme0n1.out‚71198.77099609375 disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚71198.77099609375$ disk./.usageGB‚550.7508964538574# disk.nvme0n1.in‚145023.25390625 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 669.26171875% proc.memory.availableMB‚ 333.40625 network.sent‚ 3.0179119e+07 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚145023.25390625 proc.memory.rssMB‚ 2811.03125) proc.memory.percent‚17.15717315673828 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-1.out‚ 669.26171875 cpu‚26.00171491868226 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 4.241123e+06% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.sda.out‚ 669.26171875# disk.dm-0.out‚71198.77099609375Š¶*À‘â9‚(:† ÷¬èÒöî·£ gpu.0.gpu‚9.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!zÍ£ã9‚(:” ÷¬èÒ£ û¥% disk.nvme0n1.out‚71208.4873046875 disk.sda1.in‚0 disk.dm-1.out‚ 669.65234375" disk.dm-0.out‚71208.4873046875( proc.memory.availableMB‚ 333.40234375 cpu‚26.00146254661479 disk.sda1.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.sda2.out‚0 disk.rbd1.in‚0.0625 proc.memory.rssMB‚ 2811.03125) disk./.usagePercent‚36.96172623873239$ disk./.usageGB‚550.7564315795898 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 669.65234375! disk.rbd1.out‚123.78173828125 proc.cpu.threads‚17 network.recv‚ 4.244596e+06 disk.rbd3.in‚0 network.sent‚ 3.0184204e+07 disk.rbd0.out‚0 disk.sda.out‚ 669.65234375 disk.dm-1.in‚ 0.0078125 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚71208.4873046875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-0.in‚145023.25390625) proc.memory.percent‚17.15717315673828 disk.drbd2165.out‚0$ memory_percent‚97.96507358551025‡îóÆ4ä9/ _wandbruntime‚16083  _runtime‚16083”EÌÉå9à _runtime‚16083.073565491 _step‚1559 & lr-AdamW/pg1‚3.345508498998467e-06 & lr-AdamW/pg2‚3.345508498998467e-06 trainer/global_step‚18949 $ _timestamp‚1.784288890947912e+09*jØAÎæ9È trainer/global_step‚18949 $ _timestamp‚1.784288890947912e+09 _runtime‚16083.073565491 _step‚1559 & lr-AdamW/pg1‚3.345508498998467e-06 & lr-AdamW/pg2‚3.345508498998467e-06— süÁC4ç9/ _wandbruntime‚16083  _runtime‚16083’Ø/(è9P RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚18949 _step‚1560 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175539302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40800116 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2880942985076736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325850843 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150301562 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚654615120389632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2866119686852608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚634346821675008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚654615229676032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33192071 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7842888909490333e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150301562 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20047988204544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2880942968037376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2900990973281280 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325840864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2880942985076736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40800382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2900990614902784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20268317326848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2900990973281280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚654615229676032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33192071 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7842888909490333e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150301562 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20047988204544 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2880942968037376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2900990973281280 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325840864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2880942985076736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40800382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2900990614902784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20268317326848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2900990973281280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150301562 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚654615120389632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0˜ óN¤ð4ê9/ _wandbruntime‚16083  _runtime‚16083x¾‹ã%ë9‹N LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33193816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20048659492352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325867541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20018077318284 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚634380231821312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150309194 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚654649678540800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2901139226945024 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2881090226106368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2901139226945024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40802471 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2886284615392766 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175558345 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175548354 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33193813 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚634380250433536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325867541 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150309196 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73996287 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2866266197524480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175548354 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20049000838656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325857548 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2881090209067008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2866266180485120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20018417868286 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325857548 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20269428107264 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150309194 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚16083.688970258 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2881090226106368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73996015 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _runtime‚16083.688970258 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2881090226106368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73996015 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33193816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20048659492352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325867541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20018077318284 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚634380231821312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150309194 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚654649678540800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2901139226945024 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2881090226106368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1561 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2901139226945024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40802471 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2886284615392766 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175558345 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175548354 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33193813 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚634380250433536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325867541 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150309196 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73996287 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2866266197524480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175548354 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20049000838656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325857548 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2881090209067008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2866266180485120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20018417868286 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325857548 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20269428107264 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150309194 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150683162 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150683164 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74183345 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20319593875968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚636018240989184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150683162 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚16122.844379422 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2893513243747568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20098683706368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33279321 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40903765 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175993754 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2873445203406848 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842889307153406e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20098683706368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176003731 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2908403361886208 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2893512886158318 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20067682751470 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚636018259601408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326676916 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326686895 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74183083 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20098342367232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175993754 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33279318 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚656337853477376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150683164 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176003731 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚121012º¹Ó ù9†P LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326686895 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74183083 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20098342367232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175993754 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33279318 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚656337853477376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150683164 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176003731 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2908403720264704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656337746288128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2873445220446208 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20098342367232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150683162 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150683164 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74183345 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20319593875968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚636018240989184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150683162 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚16122.844379422 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2893513243747568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20098683706368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33279321 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40903765 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175993754 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2873445203406848 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842889307153406e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20098683706368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176003731 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2908403361886208 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2893512886158318 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20067682751470 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚636018259601408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326676916 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360œ áÈ4ú9/ _wandbruntime‚16123  _runtime‚161235÷Q'û9‹N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20320529385472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2888452260548608 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2893660765473274 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚1565 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176012797 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20099696340480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40905783 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2873591731118080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176002806 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2873591714078720 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150690796 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20068693805192 trainer/global_step‚18999 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74186846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176012797 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚636051688359936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20320617955328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74187108 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2908551973928448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326693600 % _timestamp‚1.7842889313297226e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2908551973928448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2888452260548608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20099354994176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2893660407883912 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326693600 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2888452277587968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33281066 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2888452277587968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326703593 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150690794 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40906042 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚636051669747712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150690794 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326703593 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326703593 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150690794 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40906042 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚636051669747712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150690794 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326703593 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2873591731118080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176002806 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2873591714078720 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150690796 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20068693805192 trainer/global_step‚18999 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74186846 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176012797 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚636051688359936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20320617955328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74187108 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2908551973928448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326693600 % _timestamp‚1.7842889313297226e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2908551973928448 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2888452260548608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20099354994176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2893660407883912 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 Û6 4ý9/ _wandbruntime‚16123  _runtime‚16123²¿lø þ9‡ _runtime‚16123.456578312 2loss/SRqqt1/assignment_loss‚0.3196580880475687 /loss/FRt1/detection_loss‚0.5161391583248464 trainer/global_step‚18999 _step‚1566 3loss/FBt2/assignment_loss‚0.0002598871139134002 1loss/SRqqt2/detection_loss‚0.5531871809366545 .loss/FBt2/detection_loss‚0.635782950783323 epoch‚47 % _timestamp‚1.7842889313312464e+09 0loss/FRt1/assignment_loss‚1.4513274051988139 3loss/FBt1/assignment_loss‚0.0017007385019482406 1loss/SRqqt1/detection_loss‚0.8552361923523004 /loss/FBt1/detection_loss‚0.3295441413233013 %loss/total_loss‚6.936815647078138 0loss/FRt2/assignment_loss‚0.6975261372134143 1loss/SRqqt2/assignment_loss‚0.388257791146822 /loss/FRt2/detection_loss‚1.1881959761352328—±Tñÿ9Œ 1loss/SRqqt2/detection_loss‚0.5531871809366545 .loss/FBt2/detection_loss‚0.635782950783323 epoch‚47 % _timestamp‚1.7842889313312464e+09 0loss/FRt1/assignment_loss‚1.4513274051988139 3loss/FBt1/assignment_loss‚0.0017007385019482406 1loss/SRqqt1/detection_loss‚0.8552361923523004 /loss/FBt1/detection_loss‚0.3295441413233013 %loss/total_loss‚6.936815647078138 0loss/FRt2/assignment_loss‚0.6975261372134143 1loss/SRqqt2/assignment_loss‚0.388257791146822 /loss/FRt2/detection_loss‚1.1881959761352328 _runtime‚16123.456578312 2loss/SRqqt1/assignment_loss‚0.3196580880475687 /loss/FRt1/detection_loss‚0.5161391583248464 trainer/global_step‚18999 _step‚1566 3loss/FBt2/assignment_loss‚0.0002598871139134002ž õub/“€:‚(:ˆ ¤­èÒ‡žê¢ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚257.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýø^A:‚(:‘ ¤­èÒÌïठdisk.sda.in‚ 0.0078125 disk.sda2.in‚0 cpu‚26.000658577695674 disk.rbd3.in‚0# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0) disk./.usagePercent‚36.96301549267483 disk.rbd0.in‚0 proc.memory.rssMB‚ 2811.03125$ disk.nvme0n1.out‚71240.630859375 network.sent‚ 3.0257374e+07$ disk./.usageGB‚550.7756423950195 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625% disk.nvme0c0n1.in‚145023.25390625& disk.nvme0c0n1.out‚71240.630859375' proc.memory.availableMB‚ 333.2734375 disk.dm-0.in‚145023.25390625 disk.sda.out‚ 670.24609375 disk.sda2.out‚0 disk.sda3.out‚ 670.24609375) proc.memory.percent‚17.15717315673828 network.recv‚ 4.256166e+06 disk.sda3.in‚ 0.0078125 proc.cpu.threads‚17 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚123.96923828125! disk.dm-0.out‚71240.630859375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0$ memory_percent‚97.96586036682129 disk.dm-1.out‚ 670.24609375 disk.rbd3.out‚0ÝÁŒQ“‚:‚(:ˆ ³­èÒ›óÅ¢ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚259.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥çă:‚(: ³­èÒ¦Ö¯¤ disk.rbd3.in‚0& disk.nvme0n1.out‚71254.76806640625! disk.rbd1.out‚123.96923828125$ disk./.usageGB‚550.7815017700195 disk.sda2.out‚0 network.recv‚ 4.260747e+06 disk.sda3.in‚ 0.0078125# disk.dm-0.out‚71254.76806640625 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625# memory_percent‚97.9644775390625! proc.memory.availableMB‚333.5) proc.memory.percent‚17.15717315673828 disk.rbd1.in‚0.0625 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0 disk.sda.out‚ 670.55859375 disk.dm-0.in‚145023.25390625 disk.sda1.out‚0( disk.nvme0c0n1.out‚71254.76806640625 cpu‚26.000396690244845 disk.dm-1.out‚ 670.55859375 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚36.96340872024742 disk.rbd0.out‚0 network.sent‚ 3.0320546e+07 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.dm-1.in‚ 0.0078125 proc.memory.rssMB‚ 2811.03125 proc.cpu.threads‚17 disk.sda3.out‚ 670.55859375g]Òy„:‚(:„ ­èÒâÚࢠgpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X‚¿Ò…:‚(:… ­è҆ϋ¥) proc.memory.percent‚17.15717315673828 disk.sda.out‚ 670.71875 disk.sda3.out‚ 670.71875' proc.memory.availableMB‚ 333.4921875 disk.sda1.out‚0 disk.sda.in‚ 0.0078125 disk.dm-0.in‚145023.25390625 disk.rbd3.out‚0 disk.dm-0.out‚71266.05078125 network.recv‚ 4.263884e+06* disk./.usagePercent‚36.963833436746725 disk.sda1.in‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda3.in‚ 0.0078125 network.sent‚ 3.0325583e+07 disk.rbd3.in‚0 proc.cpu.threads‚17# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125 cpu‚26.00014700266536# disk.nvme0n1.out‚71266.05078125% disk.nvme0c0n1.out‚71266.05078125 disk.sda2.in‚0 disk.dm-1.out‚ 670.71875 disk.rbd1.in‚0.0625! disk.rbd1.out‚124.03173828125$ memory_percent‚97.96452522277832 proc.memory.rssMB‚ 2811.03ojÁ>125$ disk./.usageGB‚550.7878303527832 disk.rbd0.out‚0ME¸…4†:/ _wandbruntime‚16162  _runtime‚16162Mžuʇ:Ä & lr-AdamW/pg2‚2.740336945076238e-06 trainer/global_step‚19049 % _timestamp‚1.7842889707459624e+09 _runtime‚16162.871561174 _step‚1567 & lr-AdamW/pg1‚2.740336945076238e-06·7{ψ:É trainer/global_step‚19049 % _timestamp‚1.7842889707459624e+09 _runtime‚16162.871561174 _step‚1567 & lr-AdamW/pg1‚2.740336945076238e-06 & lr-AdamW/pg2‚2.740336945076238e-06Ÿ ªÑy4‰:/ _wandbruntime‚16162  _runtime‚16162·“(Š:P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41007114 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2900889036238826 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2895667088039936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2915816467248128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176448206 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20118639788268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176458183 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151064762 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327522947 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚637689697527808 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327512968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚637689678915584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2895667071000576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20149037869056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20149037869056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2895667088039936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33366568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842889707470121e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚658060291858944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _runtime‚16162.872784875 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2915816467248128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176458183 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20370703617536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2900889393828076 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20149379208192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74373682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33366571 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2915816108869632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2880770754039808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20149379208192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2915816108869632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2880770737000448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20370612943360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20118299238378 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1568 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151064762 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088?[I (‹:†P aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2880770754039808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20149379208192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2915816108869632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2880770737000448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20370612943360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20118299238378 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _step‚1568 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151064762 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41007114 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2900889036238826 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2895667088039936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2915816467248128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176448206 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20118639788268 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176458183 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151064762 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327522947 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚637689697527808 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327512968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚637689678915584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2895667071000576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20149037869056 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20149037869056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2895667088039936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33366568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7842889707470121e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚658060291858944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _runtime‚16162.872784875 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2915816467248128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176458183 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20370703617536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2900889393828076 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20149379208192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74373682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33366571 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2915816108869632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269  ÿ”+Å4Œ:/ _wandbruntime‚16163  _runtime‚16163ù(¼':ŠN F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2880917247672320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74377781 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20150391842304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33368313 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20150050496000 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784288971358666e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327539645 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74377533 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2895814312030208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151072394 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176457258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚658094847249408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚637723126286336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2915964720911872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚19049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327529652 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20150391842304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20150050496000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41009220 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚637723107674112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20371634490368 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20119650842102 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176467249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20371720963072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2901036915553782 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚658094742164480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2895814329069568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327529652 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2880917264711680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327539645 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151072394 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784288971358666e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327539645 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74377533 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2895814312030208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151072394 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176457258 OGDeviceStatsMonitor.o‚clñ n_train_batch_end/inactive_split.small_pool.current‚245 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚658094847249408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚637723126286336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2915964720911872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚19049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327529652 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20150391842304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20150050496000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41009220 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚637723107674112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20371634490368 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20119650842102 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176467249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20371720963072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚1569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2901036915553782 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚658094742164480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2895814329069568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327529652 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2880917264711680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327539645 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151072394 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2880917247672320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74377781 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664¡ †@4:/ _wandbruntime‚16163  _runtime‚16163~::Œ 0loss/FRt1/assignment_loss‚1.4662609036861514 3loss/SRqqt1/assignment_loss‚0.26576847671984527 /loss/FRt1/detection_loss‚0.5016120838096028 0loss/FBt2/detection_loss‚0.33083552219805396 %loss/total_loss‚6.543065154051033 0loss/FRt2/assignment_loss‚0.7703570370514908 /loss/FRt2/detection_loss‚1.1560538253777206 _step‚1570 3loss/SRqqt2/assignment_loss‚0.36689092252593475 trainer/global_step‚19049 2loss/FBt1/assignment_loss‚0.008988876872346233 3loss/FBt2/assignment_loss‚3.898017393460781e-05 1loss/SRqqt1/detection_loss‚0.8267745901485056 1loss/SRqqt2/detection_loss‚0.5572042702892476 0loss/FBt1/detection_loss‚0.29227966519819903 epoch‚47 % _timestamp‚1.7842889713611085e+09 _runtime‚16163.486454487êA{‘:‘ 2loss/FBt1/assignment_loss‚0.008988876872346233 3loss/FBt2/assignment_loss‚3.898017393460781e-05 1loss/SRqqt1/detection_loss‚0.8267745901485056 1loss/SRqqt2/detection_loss‚0.5572042702892476 0loss/FBt1/detection_loss‚0.29227966519819903 epoch‚47 % _timestamp‚1.7842889713611085e+09 _runtime‚16163.486454487 0loss/FRt1/assignment_loss‚1.4662609036861514 3loss/SRqqt1/assignment_loss‚0.26576847671984527 /loss/FRt1/detection_loss‚0.5016120838096028 0loss/FBt2/detection_loss‚0.33083552219805396 %loss/total_loss‚6.543065154051033 0loss/FRt2/assignment_loss‚0.7703570370514908 /loss/FRt2/detection_loss‚1.1560538253777206 _step‚1570 3loss/SRqqt2/assignment_loss‚0.36689092252593475 trainer/global_step‚19049¢ W×L$“’:‚(:ˆ Ñ­èÒÀ¬±¥ gpu.0.gpu‚31.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ä?“:‚(:‰ Ñ­èÒæÑä¦ disk.sda2.in‚0 disk.dm-1.out‚671.125 disk.rbd1.in‚0.0625# disk.dm-0.out‚71279.00537109375 disk.sda1.out‚0( proc.memory.availableMB‚ 333.17578125) proc.memory.percent‚17.15717315673828 network.recv‚ 4.267388e+06 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚124.03173828125 disk.rbd3.out‚0 proc.memory.rssMB‚ 2811.03125 cpu‚25.999863686512153 disk.rbd3.in‚0 disk.drbd2165.in‚0 network.sent‚ 3.0343668e+07 disk.sda.out‚671.125# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚550.7939643859863 disk.sda3.out‚671.125 disk.dm-0.in‚145023.25390625& disk.nvme0n1.out‚71279.00537109375$ memory_percent‚97.96645641326904) disk./.usagePercent‚36.96424509686178 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 proc.cpu.threads‚17 disk.sda.in‚ 0.0078125 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚71279.00537109375—v—£ž”:‚(:“ à­èÒ‡¾É¢ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁîDM•:‚(:Ž à­èÒÊ¥) proc.memory.percent‚17.15717315673828 cpu‚25.999592245487978! disk.rbd1.out‚124.09423828125$ memory_percent‚97.96600341796875& disk.nvme0n1.out‚71288.18408203125 disk.sda2.out‚0 disk.dm-0.in‚145023.25390625 proc.cpu.threads‚17 disk.rbd1.in‚0.0625( disk./.usagePercent‚36.9646273161104# disk.nvme0n1.in‚145023.25390625% disk.nvme0c0n1.in‚145023.25390625 disk.sda.out‚ 671.2890625 network.sent‚ 3.0393714e+07# disk.dm-0.out‚71288.18408203125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0078125" proc.memory.availableMB‚333.25 proc.memory.rssMB‚ 2811.03125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚71288.18408203125$ disk./.usageGB‚550.7996597290039 disk.rbd0.out‚0 disk.sda3.out‚ 671.2890625 disk.dm-1.out‚ 671.2890625 network.recv‚ 4.271505e+06 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.rbd3.out‚0hñá’–:‚(:‡ ï­èÒݶ…£ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚261.535' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.307 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t'ìx—:‚(:Ž ï­èÒÓª’¤ disk.sda.in‚ 0.0078125 disk.rbd3.in‚0 disk.sda3.in‚ 0.0078125 disk.rbd3.out‚0# disk.dm-0.out‚71297.18798828125( proc.memory.availableMB‚ 333.24609375$ disk./.usageGB‚550.8062858581543 disk.rbd0.in‚0! disk.rbd1.out‚124.09423828125 disk.dm-0.in‚145023.25390625 disk.sda1.out‚0% disk.nvme0c0n1.in‚145023.25390625( disk.nvme0c0n1.out‚71297.18798828125 disk.dm-1.out‚ 671.59375 cpu‚25.99934954325483 network.sent‚ 3.0398745e+07 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 671.59375 proc.memory.rssMB‚ 2811.03125& disk.nvme0n1.out‚71297.18798828125 disk.sda.out‚ 671.59375 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625# disk.nvme0n1.in‚145023.25390625$ memory_percent‚97.96602725982666) proc.memory.percent‚17.15717315673828 proc.cpu.threads‚17 network.recv‚ 4.274889e+06) disk./.usagePercent‚36.96507200119738 disk.drbd2165.in‚0 disk.sda2.in‚0 žö4˜:/ _wandbruntime‚16202  _runtime‚16202½n±·Ë™:Å ' lr-AdamW/pg1‚2.4602466284736925e-06 ' lr-AdamW/pg2‚2.4602466284736925e-06 trainer/global_step‚19099 $ _timestamp‚1.784289010699901e+09 _runtime‚16202.825566345 _step‚1571"* qК:Ê ' lr-AdamW/pg1‚2.4602466284736925e-06 ' lr-AdamW/pg2‚2.4602466284736925e-06 trainer/global_step‚19099 $ _timestamp‚1.784289010699901e+09 _runtime‚16202.825566345 _step‚1571£ TŒ!s4›:/ _wandbruntime‚16202  _runtime‚16202ÎY™E(œ:€P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33453821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2908265543908584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2908265186319334 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2903029122482176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74563873 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2923228855853056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚639361116841984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176912635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41109778 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2903029139521536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33453818 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41110052 $ _timestamp‚1.784289010701052e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151446364 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2923229214231552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74563596 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20169256275176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2903029122482176 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151446362 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20200074710016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20200074710016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20421618021376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚659782734863360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151446362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176912635 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20421712889856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176902658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2888096270594048 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2923229214231552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2923228855853056 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151446364ý·ƒR0:…P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2903029122482176 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151446362 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20200074710016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20200074710016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20421618021376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚659782734863360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151446362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176912635 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20421712889856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176902658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2888096270594048 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2923229214231552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2923228855853056 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151446364 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33453821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2908265543908584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2908265186319334 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2903029122482176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74563873 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2923228855853056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚639361116841984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176912635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41109778 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2903029139521536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33453818 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41110052 $ _timestamp‚1.784289010701052e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151446364 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2923229214231552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74563596 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20169256275176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508¤ ÛÏ4ž:/ _wandbruntime‚16203  _runtime‚16203´Ç…°'Ÿ:‹N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151453996 _runtime‚16203.438682756 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2923377467895296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328375697 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176921701 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33455566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20170267329010 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41111756 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151453996 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151453994 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151453994 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2903176363511808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74567319 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2908413065634290 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2903176380551168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2903176363511808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20201087344128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328375697 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20200745997824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20169926779008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20422636697088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2903176380551168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚639394545600512 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2888242798305280 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2908412708044928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0FØTo' :N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2903176363511808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20201087344128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328375697 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20200745997824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20169926779008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20422636697088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2903176380551168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚639394545600512 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2888242798305280 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2908412708044928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151453996 _runtime‚16203.438682756 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2923377467895296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328375697 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176921701 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33455566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20170267329010 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41111756 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151453996 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151453994 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151453994 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2903176363511808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74567319 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2908413065634290 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2903176380551168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0¥ ë_® 4¡:/ _wandbruntime‚16203  _runtime‚16203‘k;j¢:‰ 1loss/SRqqt1/detection_loss‚0.8234114365021621 1loss/SRqqt2/detection_loss‚0.5373537819790729 %loss/total_loss‚6.745735587989942 0loss/FRt2/assignment_loss‚0.7149335939279744 2loss/SRqqt2/assignment_loss‚0.4121144910237562 3loss/FBt1/assignment_loss‚0.0011116275895621968 4loss/FBt2/assignment_loss‚0.00017103286383765044 % _timestamp‚1.7842890113156846e+09 _runtime‚16203.441018977 2loss/SRqqt1/assignment_loss‚0.2767063588746472 /loss/FRt1/detection_loss‚0.4872747652392955 /loss/FBt1/detection_loss‚0.4140737720732906 /loss/FBt2/detection_loss‚0.4257763756055234 trainer/global_step‚19099 _step‚1574 epoch‚47 0loss/FRt1/assignment_loss‚1.4884842530118434 .loss/FRt2/detection_loss‚1.1643240992989763|eÖ£:Ž /loss/FBt1/detection_loss‚0.4140737720732906 /loss/FBt2/detection_loss‚0.4257763756055234 trainer/global_step‚19099 _step‚1574 epoch‚47 0loss/FRt1/assignment_loss‚1.4884842530118434 .loss/FRt2/detection_loss‚1.164324099298976 1loss/SRqqt1/detection_loss‚0.8234114365021621 1loss/SRqqt2/detection_loss‚0.5373537819790729 %loss/total_loss‚6.745735587989942 0loss/FRt2/assignment_loss‚0.7149335939279744 2loss/SRqqt2/assignment_loss‚0.4121144910237562 3loss/FBt1/assignment_loss‚0.0011116275895621968 4loss/FBt2/assignment_loss‚0.00017103286383765044 % _timestamp‚1.7842890113156846e+09 _runtime‚16203.441018977 2loss/SRqqt1/assignment_loss‚0.2767063588746472 /loss/FRt1/detection_loss‚0.4872747652392955¦ ÷Dù“¤:‚(:ˆ þ­èÒû÷Š£ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.536' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+¢?"¥:‚(:— þ­èÒµñý¤ network.sent‚ 3.0403936e+07 network.recv‚ 4.278188e+06# disk.dm-0.out‚71307.03466796875& disk.nvme0n1.out‚71307.03466796875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 671.87109375 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 671.87109375 disk.rbd3.out‚0$ memory_percent‚97.96826839447021 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0# disk.nvme0n1.in‚145023.25390625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚71307.03466796875( proc.memory.availableMB‚ 332.87890625 disk.dm-0.in‚145023.25390625) proc.memory.percent‚17.15717315673828 cpu‚25.999085958753376) disk./.usagePercent‚36.96543988402408 disk.rbd1.in‚0.0625% disk.nvme0c0n1.in‚145023.25390625 disk.sda.out‚ 671.87109375 proc.memory.rssMB‚ 2811.03125# disk./.usageGB‚550.811767578125 disk.sda1.in‚0 disk.sda1.out‚0! disk.rbd1.out‚124.15673828125 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0078125 proc.cpu.threads‚17õ¶)¦:‚(:’ ®èÒæÐ’¢ gpu.0.gpu‚50.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úb‡"§:‚(:— ®èÒ¹²‰¤$ disk./.usageGB‚550.8074607849121# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2811.03125 cpu‚25.998844082017744 network.sent‚ 3.0467039e+07 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.rbd0.out‚0 disk.rbd1.in‚0.0625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚97.96686172485352& proc.memory.availableMB‚ 333.109375) proc.memory.percent‚17.15717315673828 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.in‚145023.25390625* disk./.usagePercent‚36.965150851517926 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 671.99609375( disk.nvme0c0n1.out‚71320.77978515625! disk.rbd1.out‚124.15673828125 disk.dm-0.in‚145023.25390625 network.recv‚ 4.282518e+06& disk.nvme0n1.out‚71320.77978515625 disk.sda.out‚ 671.99609375 disk.sda2.out‚0 disk.dm-1.out‚ 671.99609375# disk.dm-0.out‚71320.77978515625©ÊÂð4¨:/ _wandbruntime‚16242  _runtime‚16242êXÌ©:Æ ' lr-AdamW/pg1‚2.1951775928915227e-06 ' lr-AdamW/pg2‚2.1951775928915227e-06 trainer/global_step‚19149 % _timestamp‚1.7842890504485283e+09 _runtime‚16242.574127395 _step‚1575;¡S:Ѫ:Ë trainer/global_step‚19149 % _timestamp‚1.7842890504485283e+09 _runtime‚16242.574127395 _step‚1575 ' lr-AdamW/pg1‚2.1951775928915227e-06 ' lr-AdamW/pg2‚2.1951775928915227e-06§ ]Øðu4«:/ _wandbruntime‚16242  _runtime‚16242¯<œ{Š¬:€P D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641032554768384 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2915641693989092 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚19149 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2930641602836480 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151827964 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20472717921792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2895421821227008 % _timestamp‚1.7842890504496562e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41212618 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33541071 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74753686 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16242.57550183 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177357110 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2930641602836480 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151827964 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177357110 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641032573380608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2910391191003136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20250770211840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983#š¦œ (­:…P UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2915641336399842 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151827962 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177367087 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33541068 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2930641961214976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20250770211840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177357110 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2930641602836480 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151827964 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177357110 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641032573380608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚1576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2910391191003136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20250770211840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641032554768384 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2915641693989092 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚19149 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2930641602836480 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151827964 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20472717921792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2895421821227008 % _timestamp‚1.7842890504496562e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41212618 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33541071 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74753686 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚16242.57550183 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2910538432032768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2915789215714798 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20220883815918 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2895568331898880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20251441499648 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74757476 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2895568314859520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33542816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20251441499648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177366162 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20220543265916 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177366162 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2910538414993408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20251782845952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚19149 _step‚1577 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2930789856493056 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151835596 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33542813 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20473740481536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2910538414993408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7842890510611784e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74757756 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41214663 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329211749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329201756 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20251782845952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151835594 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641066002139136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2930790214878720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚661539742620672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20473641425920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329211749 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚661539624952832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177376153 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151835594 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177376153 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2930790214878720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641065983526912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2910538432032768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2915789215714798 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20220883815918 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2895568331898880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20251441499648 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74757476 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2895568314859520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33542816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20251441499648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177366162 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20220543265916 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177366162 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2910538414993408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20251782845952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚19149 _step‚1577 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2930789856493056 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151835596 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33542813 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20473740481536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2910538414993408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7842890510611784e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74757756 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41214663 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329211749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329201756 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20251782845952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151835594 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641066002139136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2930790214878720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚661539742620672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20473641425920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329211749 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚661539624952832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177376153 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151835594 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177376153 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2930790214878720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641065983526912 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20301124374528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177821539 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2938054349819904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2917753225445376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚663227739713536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚642703992694784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2902747354820608 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152209562 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2917753225445376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2938054708198400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33628321 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2902747337781248 % _timestamp‚1.7842890904186254e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177811562 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 330021124 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152209564 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177811562 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41315748 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚663227622038528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚642704011307008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20270489248992 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2938054708198400 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20301124374528 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33628318 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚16282.547374478 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74943786 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177821539 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2917753242484736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚642704011307008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20270489248992 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2938054708198400 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20301124374528 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33628318 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚16282.547374478 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74943786 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177821539 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2917753242484736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20301124374528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177821539 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2938054349819904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2917753225445376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚663227739713536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚642703992694784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2902747354820608 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152209562 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2917753225445376 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2938054708198400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33628321 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2902747337781248 % _timestamp‚1.7842890904186254e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177811562 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888¬ j2Ê4À:/ _wandbruntime‚16283  _runtime‚16283tìÝÆ Á:‹N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2917900466475008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74947882 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚16283.158701303 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177830605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663262187841024 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2902893848453120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330047801 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74947625 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20302478347776 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2923165008205944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152217196 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2917900466475008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33630066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20271500302826 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330037808 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152217196 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642737440065536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚663262086950400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642737421453312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2917900483514368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330047801 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20524747775488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚19199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20302137001472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330037808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33630063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¥Wžâ'Â:N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20302137001472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330037808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33630063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2917900466475008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74947882 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚16283.158701303 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177830605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663262187841024 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2902893848453120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330047801 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74947625 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20302478347776 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2923165008205944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152217196 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2917900466475008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33630066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20271500302826 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330037808 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152217196 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642737440065536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚663262086950400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642737421453312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2917900483514368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330047801 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20524747775488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 trainer/global_step‚19199­ žO4Ã:/ _wandbruntime‚16283  _runtime‚16283¿ÃOÄ:Š 2loss/FBt2/assignment_loss‚0.012274718951014018 /loss/FRt1/detection_loss‚0.4889847349960011 1loss/SRqqt1/detection_loss‚0.6742827788908035 &loss/total_loss‚6.8174248097566235 epoch‚47 _step‚1582 0loss/FRt1/assignment_loss‚1.3844729455844238 3loss/SRqqt1/assignment_loss‚0.42219659578075314 3loss/FBt1/assignment_loss‚0.0010419821178919207 /loss/FBt1/detection_loss‚0.3496358975639319 trainer/global_step‚19199 0loss/FRt2/assignment_loss‚0.7698462480228345 % _timestamp‚1.7842890910351582e+09 _runtime‚16283.160517872 2loss/SRqqt2/assignment_loss‚0.4310039971508248 /loss/FRt2/detection_loss‚1.1894643237482534 1loss/SRqqt2/detection_loss‚0.5994847883098442 /loss/FBt2/detection_loss‚0.49473579864004730ú©øÅ: 2loss/SRqqt2/assignment_loss‚0.4310039971508248 /loss/FRt2/detection_loss‚1.1894643237482534 1loss/SRqqt2/detection_loss‚0.5994847883098442 /loss/FBt2/detection_loss‚0.4947357986400473 2loss/FBt2/assignment_loss‚0.012274718951014018 /loss/FRt1/detection_loss‚0.4889847349960011 1loss/SRqqt1/detection_loss‚0.6742827788908035 &loss/total_loss‚6.8174248097566235 epoch‚47 _step‚1582 0loss/FRt1/assignment_loss‚1.3844729455844238 3loss/SRqqt1/assignment_loss‚0.42219659578075314 3loss/FBt1/assignment_loss‚0.0010419821178919207 /loss/FBt1/detection_loss‚0.3496358975639319 trainer/global_step‚19199 0loss/FRt2/assignment_loss‚0.7698462480228345 % _timestamp‚1.7842890910351582e+09 _runtime‚16283.160517872® æ{ì‘Æ:‚(:† É®èÒÄÃ⣠gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚286.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z;ÁÇ:‚(:’ É®èÒ¦¦š¥ disk.sda2.in‚0! disk.rbd1.out‚124.34423828125* proc.memory.percent‚17.161178588867188 network.recv‚ 4.296247e+06( disk.nvme0c0n1.out‚71367.05908203125 disk.sda3.out‚ 673.265625 disk.rbd3.in‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 673.265625( proc.memory.availableMB‚ 332.20703125 cpu‚25.999364902773824 disk.dm-0.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.sda2.out‚0 disk.rbd1.in‚0.0625 proc.memory.rssMB‚ 2811.6875# disk.dm-0.out‚71367.05908203125 disk.sda1.in‚0 disk.sda.out‚ 673.265625$ memory_percent‚97.97236919403076$ disk./.usageGB‚550.8426208496094 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625 disk.sda1.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.in‚145023.25390625 network.sent‚ 3.0544508e+07) disk./.usagePercent‚36.96751047296102& disk.nvme0n1.out‚71367.05908203125 disk.drbd2165.in‚0xL$4È:/ _wandbruntime‚16300  _runtime‚16300r&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2727902829647064 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 particle/accuracy_4_of_5‚1 epoch‚47 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970836400985718 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021032636985182762 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45287632942199707 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047128789126873016 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5724201798439026 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25239747762680054 (particle/accuracy‚0.8415411114692688 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.573492169380188 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3558109700679779 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6588345766067505 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22035641968250275 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6433124542236328 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_2_of_3‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_2‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27422577142715454 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7611598968505859 +particle/sensitivity‚0.6146659255027771 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25030073523521423 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5422549843788147 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8008456230163574 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6096518039703369 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5322744846343994 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043487269431352615 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 particle/accuracy_3_of_4‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942677617073059 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5952193737030029 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6572545766830444 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +particle/specificity‚0.9035872220993042 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.210791677236557 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6321728825569153 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7945781946182251 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2715485095977783 'particle/f_score‚0.6248884201049805 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6259875893592834 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03552964702248573 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 _runtime‚16300.234340535 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9134987592697144 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5999107360839844 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20371854305267334 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23845143616199493 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8452845215797424 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5939664244651794 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_4‚0.9857255220413208 particle/accuracy_1_of_4‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6147758364677429 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07827772200107574 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6557745933532715 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037037014961243 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 *jet/accuracy_1_of_1‚0.3284560739994049 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20661962032318115 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6244534254074097 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022613270208239555 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3299621641635895 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3201545178890228 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_3_of_5‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23777377605438232 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +jet/accuracy_3_of_4‚0.07847467809915543 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7267573475837708 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37267574667930603 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6888877153396606 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35607296228408813 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_1_of_2‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5174999833106995 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037037014961243 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6594260334968567 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1981143206357956 trainer/global_step‚19199 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7296673655509949 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 6validation_average_jet_accuracy‚0.3719010353088379 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6244454979896545 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23935231566429138 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.057402677834033966 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.654110848903656 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7218757271766663 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7192723155021667 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7285914421081543 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2547062039375305 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7219009399414062 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762619495391846 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7334467172622681 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_4_of_4‚0.9510270357131958 % _timestamp‚1.7842891081080692e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26357874274253845 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6551845073699951 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6013492941856384 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4870161712169647 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2625800669193268 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5307394862174988 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.041441187262535095 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_2_of_2‚0.15794473886489868 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02115723118185997 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7759644389152527 ,jet/accuracy_4_of_4‚0.011098379269242287 “3‹?fjet/accuracy_5_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857453107833862 7"Purity/1FRt0SRqqt*FBt/event_purity‚0.27253457903862 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.266561359167099 particle/accuracy_1_of_3‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2625800669193268 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022459974512457848 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6242038011550903 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6605631113052368 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24904169142246246 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 particle/accuracy_3_of_3‚1 _step‚1583 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995073139667511 7"Purity/*FRt*SRqqt0FBt/event_purity‚0.29597008228302 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6922633647918701 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8526139855384827 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6303046941757202 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0893634632229805 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_3‚0.9253297448158264 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24344423413276672 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22469599545001984 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762619495391846 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5952374339103699 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886239171028137 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6013492941856384 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2389044463634491 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6217002868652344 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7645206451416016 *jet/accuracy_2_of_3‚0.5727574825286865 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 *jet/accuracy_2_of_4‚0.8052418828010559 jet/accuracy_2_of_5‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.586763322353363 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22576870024204254 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28651341795921326 ("Purity/2FRt2SRqqt*FBt/event_purity‚0ÿ@¬=Ê:Š{ =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 +particle/specificity‚0.9035872220993042 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.210791677236557 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6321728825569153 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7945781946182251 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2715485095977783 'particle/f_score‚0.6248884201049805 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6259875893592834 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03552964702248573 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 _runtime‚16300.234340535 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9134987592697144 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5999107360839844 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20371854305267334 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23845143616199493 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8452845215797424 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5939664244651794 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_4‚0.9857255220413208 particle/accuracy_1_of_4‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6147758364677429 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07827772200107574 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6557745933532715 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037037014961243 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 *jet/accuracy_1_of_1‚0.3284560739994049 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20661962032318115 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6244534254074097 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022613270208239555 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3299621641635895 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3201545178890228 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_3_of_5‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23777377605438232 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +jet/accuracy_3_of_4‚0.07847467809915543 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7267573475837708 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.37267574667930603 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6888877153396606 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35607296228408813 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_1_of_2‚1 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5174999833106995 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037037014961243 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6594260334968567 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1981143206357956 trainer/global_step‚19199 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7296673655509949 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 6validation_average_jet_accuracy‚0.3719010353088379 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6244454979896545 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23935231566429138 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.057402677834033966 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.654110848903656 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7218757271766663 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7192723155021667 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7285914421081543 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2547062039375305 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7219009399414062 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762619495391846 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7334467172622681 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_4_of_4‚0.9510270357131958 % _timestamp‚1.7842891081080692e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26357874274253845 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6551845073699951 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6013492941856384 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4870161712169647 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2625800669193268 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5307394862174988 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.041441187262535095 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_2_of_2‚0.15794473886489868 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02115723118185997 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7759644389152527 ,jet/accuracy_4_of_4‚0.011098379269242287 jet/accuracy_5_of_5‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857453107833862 7"Purity/1FRt0SRqqt*FBt/event_purity‚0.27253457903862 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.266561359167099 particle/accuracy_1_of_3‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2625800669193268 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022459974512457848 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9895361661911011 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6242038011550903 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6605631113052368 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24904169142246246 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 particle/accuracy_3_of_3‚1 _step‚1583 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995073139667511 7"Purity/*FRt*SRqqt0FBt/event_purity‚0.29597008228302 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6922633647918701 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8526139855384827 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6303046941757202 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0893634632229805 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_3‚0.9253297448158264 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24344423413276672 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22469599545001984 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762619495391846 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5952374339103699 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886239171028137 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6013492941856384 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2389044463634491 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6217002868652344 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7645206451416016 *jet/accuracy_2_of_3‚0.5727574825286865 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 *jet/accuracy_2_of_4‚0.8052418828010559 jet/accuracy_2_of_5‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.586763322353363 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22576870024204254 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28651341795921326 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.546251654624939 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.953703761100769 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7686429619789124 *jet/accuracy_1_of_2‚0.7878735661506653 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21879561245441437 )loss/val_total_loss‚8.091022590592862 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5834892988204956 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6210771799087524 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2727902829647064 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 particle/accuracy_4_of_5‚1 epoch‚47 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970836400985718 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021032636985182762 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45287632942199707 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047128789126873016 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5724201798439026 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25239747762680054 (particle/accuracy‚0.8415411114692688 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.573492169380188 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3558109700679779 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6588345766067505 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22035641968250275 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6433124542236328 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_2_of_3‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_2‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27422577142715454 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7611598968505859 +particle/sensitivity‚0.6146659255027771 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894636273384094 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25030073523521423 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5422549843788147 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8008456230163574 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6096518039703369 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5322744846343994 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043487269431352615 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 particle/accuracy_3_of_4‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9942677617073059 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5952193737030029 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6572545766830444 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697¯ ˆóW“Ë:‚(:ˆ Ø®èÒ£œï¢ gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚261.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 õ‹Ì:‚(:Œ Ø®èÒ±ˆ‚¥$ disk.nvme0c0n1.out‚ 71375.3984375 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125! disk.rbd1.out‚124.40673828125 disk.rbd3.out‚0* proc.memory.percent‚17.167115211486816 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda2.in‚0 cpu‚26.00169017879952$ disk./.usageGB‚550.8481903076172 disk.sda1.in‚0 disk.sda3.out‚ 674.0546875 disk.rbd1.in‚0.0625 disk.dm-0.in‚145023.25390625# proc.memory.rssMB‚ 2812.66015625 network.recv‚ 4.300371e+06 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.sda.out‚ 674.0546875 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0 disk.dm-0.out‚ 71375.3984375$ memory_percent‚97.97804355621338 proc.cpu.threads‚17 network.sent‚ 3.0607576e+07" disk.nvme0n1.out‚ 71375.3984375) disk./.usagePercent‚36.96788424396101( proc.memory.availableMB‚ 331.27734375# disk.nvme0n1.in‚145023.25390625 disk.dm-1.out‚ 674.0546875 disk.sda1.out‚0 disk.rbd0.in‚0ëaiÍ:‚(:“ ç®èÒÃë¹¥) disk./.usagePercent‚36.96834454550757 disk.dm-1.in‚ 0.0078125* proc.memory.percent‚17.167115211486816 network.sent‚ 3.0664868e+07% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.sda.out‚ 674.390625 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625& proc.memory.availableMB‚ 331.640625# disk.nvme0n1.in‚145023.25390625( disk.nvme0c0n1.out‚71388.88134765625 disk.dm-1.out‚ 674.390625 cpu‚26.00142755195439 disk.drbd2165.in‚0# disk.dm-0.out‚71388.88134765625$ memory_percent‚97.97582626342773 disk.rbd3.out‚0# proc.memory.rssMB‚ 2812.66015625 proc.cpu.threads‚17$ disk./.usageGB‚550.8550491333008 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 network.recv‚ 4.304706e+06& disk.nvme0n1.out‚71388.88134765625 disk.sda3.out‚ 674.390625! disk.rbd1.out‚124.40673828125 disk.dm-0.in‚145023.25390625 disk.sda1.in‚0ݼDÝžÎ:‚(:“ ç®èÒÊ•÷® gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚255.879' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.175799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v(ÚžÏ:‚(:“ ö®èÒ⛵¢ gpu.0.gpu‚59.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚282.824' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.564800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û =$Ð:‚(:™ ö®èÒÚȵ¤% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.rbd1.in‚0.0625' proc.memory.availableMB‚ 331.1171875 disk.sda2.in‚0! disk.rbd1.out‚124.46923828125 disk.rbd3.in‚0# disk.dm-0.out‚71403.82275390625& disk.nvme0n1.out‚71403.82275390625* proc.memory.percent‚17.167115211486816 cpu‚26.001185430668762 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 674.95703125) disk./.usagePercent‚36.96865124253359 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0# memory_percent‚97.9790210723877 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 674.95703125# disk./.usageGB‚550.859619140625 disk.sda.out‚ 674.95703125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚145023.25390625# disk.nvme0n1.in‚145023.25390625 disk.rbd0.in‚0 network.sent‚ 3.0669946e+07 network.recv‚ 4.307982e+06( disk.nvme0c0n1.out‚71403.82275390625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2812.66015625ÅeÍ4Ñ:/ _wandbruntime‚16339  _runtime‚16339¦lgÌÒ:Æ ' lr-AdamW/pg1‚1.7101704984613016e-06 ' lr-AdamW/pg2‚1.7101704984613016e-06 trainer/global_step‚19249 % _timestamp‚1.7842891475072255e+09 _runtime‚16339.632846997 _step‚15843%PbÑÓ:Ë _step‚1584 ' lr-AdamW/pg1‚1.7101704984613016e-06 ' lr-AdamW/pg2‚1.7101704984613016e-06 trainer/global_step‚19249 % _timestamp‚1.7842891475072255e+09 _runtime‚16339.632846997° "ôB™4Ô:/ _wandbruntime‚16339  _runtime‚16339+&J‚‚ Õ:€P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20369553081856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664984439029760 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664984340229120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1585 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20338484911239 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20369894420992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152711828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7842891475083034e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20369894420992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75167952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644391361333248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2947414396659200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2911981551628288 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152711828 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2927044860616704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331058143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20593077696512 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1585 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20338484911239 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20369894420992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152711828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 % _timestamp‚1.7842891475083034e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20369894420992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75167952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644391361333248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2947414396659200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2911981551628288 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152711828 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2927044860616704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331058143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20593077696512 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20369553081856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664984439029760 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664984340229120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20594103690752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚665018790794752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚665018893782528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331064848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331074841 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75171809 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20339495964961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19249 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2932467898815123 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41452232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20594019315200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20370907055104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2927192084606976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152719462 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20370565708800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152719462 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178345388 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20370565708800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2927192084606976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331074841 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842891481253612e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2947562650315776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2927192101646336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75172068 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178345388 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33719836 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33719833 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2947563008701440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644424790091776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331064848 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331064848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331074841 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75171809 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20339495964961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19249 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2932467898815123 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41452232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20594019315200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20370907055104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2927192084606976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152719462 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20370565708800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152719462 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178345388 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20370565708800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2927192084606976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331074841 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7842891481253612e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2947562650315776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚1586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2927192101646336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75172068 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178345388 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33719836 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33719833 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2947563008701440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644424790091776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331064848 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20594103690752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚665018790794752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚665018893782528² ÎyF4Ú:/ _wandbruntime‚16340  _runtime‚16340YƒTü±Û:‰ % _timestamp‚1.7842891481276155e+09 _runtime‚16340.252937323 2loss/SRqqt1/assignment_loss‚0.3237495499310772 1loss/SRqqt2/detection_loss‚0.6400122048917884 epoch‚48 _step‚1587 0loss/FRt1/assignment_loss‚1.4502449702430558 2loss/SRqqt2/assignment_loss‚0.4111857618078664 1loss/SRqqt1/detection_loss‚0.7475072889993329 /loss/FBt2/detection_loss‚0.6082077910262645 %loss/total_loss‚6ö${&^.887366435979676 trainer/global_step‚19249 0loss/FRt2/assignment_loss‚0.6980834277793491 3loss/FBt1/assignment_loss‚0.0033400109029905144 4loss/FBt2/assignment_loss‚6.1072311543360534e-06 /loss/FRt1/detection_loss‚0.4980447922995134 .loss/FRt2/detection_loss‚1.249769474708076 /loss/FBt1/detection_loss‚0.2572150561592073Á>öìÜ:Ž 0loss/FRt2/assignment_loss‚0.6980834277793491 3loss/FBt1/assignment_loss‚0.0033400109029905144 4loss/FBt2/assignment_loss‚6.1072311543360534e-06 /loss/FRt1/detection_loss‚0.4980447922995134 .loss/FRt2/detection_loss‚1.249769474708076 /loss/FBt1/detection_loss‚0.2572150561592073 % _timestamp‚1.7842891481276155e+09 _runtime‚16340.252937323 2loss/SRqqt1/assignment_loss‚0.3237495499310772 1loss/SRqqt2/detection_loss‚0.6400122048917884 epoch‚48 _step‚1587 0loss/FRt1/assignment_loss‚1.4502449702430558 2loss/SRqqt2/assignment_loss‚0.4111857618078664 1loss/SRqqt1/detection_loss‚0.7475072889993329 /loss/FBt2/detection_loss‚0.6082077910262645 %loss/total_loss‚6.887366435979676 trainer/global_step‚19249³ H—«“Ý:‚(:ˆ …¯èÒéã‚£ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚240.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÓ­!Þ:‚(:– …¯èÒ¬êü¤ disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 network.sent‚ 3.0675e+07 network.recv‚ 4.311378e+06 disk.drbd2165.in‚0 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 676.9921875# disk.dm-0.out‚71423.29833984375$ memory_percent‚97.97940254211426* disk./.usagePercent‚36.969328894476725$ disk./.usageGB‚550.8697166442871 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 676.9921875 disk.dm-1.in‚ 0.0078125* proc.memory.percent‚17.167115211486816 cpu‚26.000908036168408# disk.nvme0n1.in‚145023.25390625( disk.nvme0c0n1.out‚71423.29833984375 disk.sda.in‚ 0.0078125 disk.rbd1.in‚0.0625# proc.memory.rssMB‚ 2812.66015625! disk.rbd1.out‚124.46923828125& disk.nvme0n1.out‚71423.29833984375 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda.out‚ 676.9921875 disk.dm-0.in‚145023.25390625' proc.memory.availableMB‚ 331.0546875 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0˜ñòœžß:‚(:“ ”¯èÒлͣ gpu.0.gpu‚89.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßm×&à:‚(:› ”¯èÒ—ä* disk./.usagePercent‚36.969539076662336 disk.dm-0.in‚145023.25390625 disk.sda2.in‚0! disk.rbd1.out‚124.53173828125 disk.rbd3.out‚0$ memory_percent‚97.97847270965576 network.recv‚ 4.31602e+06# disk.dm-0.out‚71435.49560546875 disk.rbd0.out‚0 disk.sda.out‚ 679.44921875 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 679.44921875( proc.memory.availableMB‚ 331.20703125$ disk./.usageGB‚550.8728485107422& disk.nvme0n1.out‚71435.49560546875 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚71435.49560546875 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 679.44921875* proc.memory.percent‚17.167115211486816 proc.cpu.threads‚17 network.sent‚ 3.0738236e+07 disk.sda1.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0 disk.sda3.in‚ 0.0078125# proc.memory.rssMB‚ 2812.66015625 cpu‚26.00066495082707v‘´Ë4á:/ _wandbruntime‚16379  _runtime‚16379äˆô¡Ìâ:Æ % _timestamp‚1.7842891874435537e+09 _runtime‚16379.569156411 _step‚1588 ' lr-AdamW/pg1‚1.4902635907126364e-06 ' lr-AdamW/pg2‚1.4902635907126364e-06 trainer/global_step‚19299ÛÛšbÑã:Ë % _timestamp‚1.7842891874435537e+09 _runtime‚16379.569156411 _step‚1588 ' lr-AdamW/pg1‚1.4902635907126364e-06 ' lr-AdamW/pg2‚1.4902635907126364e-06 trainer/global_step‚19299´ + “Ÿ4ä:/ _wandbruntime‚16379  _runtime‚16379-õþ(å:P ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2919307068182528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚19299 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153093428 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2954827502021120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20420248583680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _runtime‚16379.570466133 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75358619 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153093430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666707176115712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178800765 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2954827143642624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41553281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41553543 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚646062780647424 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2934406895058944 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20420589922816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20644484045312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178800765 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1589 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2954827502021120 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2934406912098304 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666707283304960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2954827143642624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20389441948037 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20420248583680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331894195 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2939696169580675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75358884 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2919307085221888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2934406912098304 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666707283304960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2954827143642624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20389441948037 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20420248583680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331894195 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2939696169580675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75358884 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2919307085221888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚19299 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153093428 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2954827502021120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20420248583680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _runtime‚16379.570466133 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75358619 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153093430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666707176115712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178800765 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2954827143642624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41553281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41553543 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚646062780647424 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2934406895058944 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438µ `äý‘ç:‚(:† £¯èÒÝßÆ¢ gpu.0.gpu‚89.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚231.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔârÐè:‚(:“ £¯èÒ„¦Ô¥' disk.nvme0c0n1.out‚71449.6787109375* disk./.usagePercent‚36.969986065817125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125* proc.memory.percent‚17.167115211486816 proc.cpu.threads‚17% disk.nvme0n1.out‚71449.6787109375% disk.nvme0c0n1.in‚145023.25390625 network.recv‚ 4.319635e+06# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚71449.6787109375# proc.memory.rssMB‚ 2812.66015625 disk.sda.out‚ 679.921875! disk.rbd1.out‚124.53173828125 disk.rbd3.out‚0# disk./.usageGB‚550.879508972168 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125 disk.sda2.in‚0$ memory_percent‚97.98147678375244( proc.memory.availableMB‚ 330.71484375 disk.sda1.out‚0 disk.sda3.out‚ 679.921875 disk.dm-1.out‚ 679.921875 disk.rbd1.in‚0.0625 cpu‚26.000401840354158 network.sent‚ 3.0756417e+07 disk.dm-0.in‚145023.25390625h$´4é:/ _wandbruntime‚16380  _runtime‚16380ÅÕX«ê:ŠN E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153101062 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2934554136088576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2939844048895631 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2939843691306269 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20390453001871 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178809831 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚646096209405952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20421261210624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7842891880570703e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41555294 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚19299 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33807086 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666741638387712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20645509163008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331900900 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153101060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚16380.182906204 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20421602556928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178799840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33807083 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2954975755684864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2934554136088576 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2919453578854400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178799840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41555539 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666741737181184 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2919453595893760 _step‚1590 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20421602556928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331910893 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153101060 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178809831 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331910893 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75362377 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20390112451869 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2934554153127936 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2954975397299200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75362625 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2934554153127936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20421261210624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20645428981760 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2934554136088576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2939844048895631 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2939843691306269 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20390453001871 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178809831 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚646096209405952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20421261210624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7842891880570703e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41555294 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚19299 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33807086 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666741638387712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20645509163008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331900900 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153101060 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚16380.182906204 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20421602556928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178799840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33807083 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2954975755684864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2934554136088576 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2919453578854400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178799840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41555539 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666741737181184 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2919453595893760 _step‚1590 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20421602556928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331910893 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153101060 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178809831 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331910893 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75362377 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20390112451869 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2934554153127936 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2954975397299200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75362625 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2934554153127936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20421261210624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20645428981760 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2962240249004544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20439717885055 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20695468435968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚16419.525834987 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2962239890626048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2941768946540544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179245240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20471285424640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41655932 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20471285424640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153475028 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚668429687009792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚668429794199040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2947072677250433 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0®%©ŠX ø:†P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚1593 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179245240 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647734237186048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2962240249004544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332730247 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332720268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20695557012992 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2962239890626048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33892591 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚647734218573824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20471285424640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153475028 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚668429687009792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚668429794199040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29470726772Ó ¡´50433 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2941768963579904 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2941768963579904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179255217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41656197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2926632618815488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179255217 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2926632601776128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2941768946540544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2962240249004544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20439717885055 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20695468435968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚16419.525834987 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2962239890626048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2941768946540544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179245240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20471285424640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41655932 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332746945 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20440728938777 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚19349 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2941916187570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153482660 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179254292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75552306 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16420.138964308 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20472298058752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2941916187570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚668464128654336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33894336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179254292 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚668464244225024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20441069488779 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888ŸmÌ'û:N QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332746945 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20440728938777 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚19349 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2941916187570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153482660 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179254292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75552306 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16420.138964308 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20472298058752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2941916187570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚668464128654336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33894336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179254292 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚668464244225024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20441069488779 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332746945 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179264283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2962388502668288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20471956712448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2941916204609536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20472298058752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332736952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33894333 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚647767665944576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2926779129487360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2926779112448000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438º ïÙë4ü:/ _wandbruntime‚16420  _runtime‚16420ö_Yý:‰ 0loss/FRt1/assignment_loss‚1.4566036964112323 2loss/SRqqt2/assignment_loss‚0.4459085410101575 2loss/FBt1/assignment_loss‚0.008311444850543396 3loss/FBt2/assignment_loss‚0.0001926257500229374 /loss/FRt2/detection_loss‚1.1612100112093946 0loss/SRqqt2/detection_loss‚0.630055776856939 0loss/FRt1/detection_loss‚0.47434888356277227 %loss/total_loss‚6.822641295971435 % _timestamp‚1.7842892280157163e+09 _step‚1595 0loss/FRt2/assignment_loss‚0.7674283353487651 3loss/SRqqt1/assignment_loss‚0.37364455429668203 0loss/FBt1/detection_loss‚0.35525195508300333 epoch‚48 trainer/global_step‚19349 0loss/SRqqt1/detection_loss‚0.757018977303453 /loss/FBt2/detection_loss‚0.3926664942884695 _runtime‚16420.141053491¬bH2þ:Ž 0loss/SRqqt1/detection_loss‚0.757018977303453 /loss/FBt2/detection_loss‚0.3926664942884695 _runtime‚16420.141053491 0loss/FRt1/assignment_loss‚1.4566036964112323 2loss/SRqqt2/assignment_loss‚0.4459085410101575 2loss/FBt1/assignment_loss‚0.008311444850543396 3loss/FBt2/assignment_loss‚0.0001926257500229374 /loss/FRt2/detection_loss‚1.1612100112093946 0loss/SRqqt2/detection_loss‚0.630055776856939 0loss/FRt1/detection_loss‚0.47434888356277227 %loss/total_loss‚6.822641295971435 % _timestamp‚1.7842892280157163e+09 _step‚1595 0loss/FRt2/assignment_loss‚0.7674283353487651 3loss/SRqqt1/assignment_loss‚0.37364455429668203 0loss/FBt1/detection_loss‚0.35525195508300333 epoch‚48 trainer/global_step‚19349» t„“ÿ:‚(:ˆ ЯèÒ¤¦ù¢ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\„€;‚(:‘ ЯèÒûö¤$ disk./.usageGB‚550.8946876525879 network.sent‚ 3.0853434e+07# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.in‚0$ memory_percent‚97.98173904418945* disk./.usagePercent‚36.971004719795864 disk.rbd1.in‚0.0625! disk.rbd1.out‚124.65673828125! disk.dm-0.out‚71482.361328125 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 681.2734375# proc.memory.rssMB‚ 2812.66015625 disk.dm-1.out‚ 681.2734375$ disk.nvme0n1.out‚71482.361328125 cpu‚25.99963039315475 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚ 4.331429e+06 disk.rbd3.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚71482.361328125 disk.sda2.in‚0* proc.memory.percent‚17.167115211486816 disk.dm-1.in‚ 0.0078125 disk.dm-0.in‚145023.25390625 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125& proc.memory.availableMB‚ 330.671875 disk.rbd0.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda.out‚ 681.2734375 disk.sda2.out‚0|:ÛØ“;‚(:ˆ ߯èÒý¨Ë¢ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯±%£"‚;‚(:— ߯èÒ‰Àठdisk.rbd3.in‚0" disk.dm-0.out‚71497.3974609375# disk.nvme0n1.in‚145023.25390625 disk.rbd0.in‚0 cpu‚25.99938733471447 disk.sda1.out‚0 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 681.51171875 disk.dm-1.in‚ 0.0078125$ disk./.usageGB‚550.9007530212402% disk.nvme0c0n1.in‚145023.25390625 disk.sda1.in‚0% disk.nvme0n1.out‚71497.3974609375* proc.memory.percent‚17.167115211486816 disk.dm-1.out‚ 681.51171875' disk.nvme0c0n1.out‚71497.3974609375 disk.dm-0.in‚145023.25390625 disk.drbd2165.in‚0$ memory_percent‚97.98076152801514( proc.memory.availableMB‚ 330.83203125 network.recv‚ 4.336069e+06( disk./.usagePercent‚36.9714117717753 disk.sda.out‚ 681.51171875 disk.sda2.in‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚124.65673828125 network.sent‚ 3.0916747e+07 disk.sda.in‚ 0.0078125 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2812.66015625 proc.cpu.threads‚17 disk.sda2.out‚0èÒo!‘ƒ;‚(:† î¯èÒ¸Öº¢ gpu.0.gpu‚50.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.62' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cÈNÿJ„;‚(:› î¯èÒá¶å¤! disk.rbd1.out‚124.71923828125& disk.nvme0n1.out‚71508.26220703125$ disk./.usageGB‚550.9061431884766( disk.nvme0c0n1.out‚71508.26220703125 disk.sda2.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 330.58203125 disk.sda1.in‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda3.out‚ 681.83203125# disk.nvme0n1.in‚145023.25390625* proc.memory.percent‚17.167115211486816 disk.dm-0.in‚145023.25390625) disk./.usagePercent‚36.97177351042119 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚ 4.339123e+06 disk.dm-1.out‚ 681.83203125$ memory_percent‚97.98228740692139 network.sent‚ 3.092178e+07 disk.dm-1.in‚ 0.0078125 disk.sda1.out‚0 disk.sda.out‚ 681.83203125 disk.rbd1.in‚0.0625# disk.dm-0.out‚71508.26220703125 cpu‚.Ò§Ü25.999130185997167 disk.rbd3.in‚0 disk.sda.in‚ 0.0078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.0078125# proc.memory.rssMB‚ 2812.66015625zk4Â4…;/ _wandbruntime‚16459  _runtime‚16459uï6̆;Æ ' lr-AdamW/pg1‚1.0957136742897513e-06 ' lr-AdamW/pg2‚1.0957136742897513e-06 trainer/global_step‚19399 % _timestamp‚1.7842892671840882e+09 _runtime‚16459.309709164 _step‚1596ux´3ч;Ë ' lr-AdamW/pg1‚1.0957136742897513e-06 ' lr-AdamW/pg2‚1.0957136742897513e-06 trainer/global_step‚19399 % _timestamp‚1.7842892671840882e+09 _runtime‚16459.309709164 _step‚1596¼ úÙQq4ˆ;/ _wandbruntime‚16459  _runtime‚16459Ý;? (‰;P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2949131015061504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2949130998022144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2969652995987968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41758255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649405656500224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2954448827330941 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2933958152409088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2933958135369728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 333566299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179709669 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚670152216719360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649405675112448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842892671851854e+09 _runtime‚16459.314376906 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20490334371963 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153856630 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153856628 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚19399 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20521980926464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2969652637609472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20490674921853 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2969652995987968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2969652637609472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20746560219136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 333556320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20521639587328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41758532 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179709669 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20521639587328 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2949130998022144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33979838 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33979841 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚670152330200064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153856628 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75738373 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179699692ˆ0% (Š;†P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179699692 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153856630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2969652637609472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20746560219136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 333556320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20521639587328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41758532 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179709669 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20521639587328 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2949130998022144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33979838 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33979841 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚670152330200064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153856628 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75738373 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179699692 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2949131015061504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2949130998022144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2969652995987968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41758255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649405656500224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2954448827330941 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2933958152409088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2933958135369728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 333566299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179709669 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚670152216719360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649405675112448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7842892671851854e+09 _runtime‚16459.314376906 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20490334371963 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153856630 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153856628 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚19399 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20521980926464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2969652637609472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20490674921853 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2969652995987968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75741878 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2954596349056647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚670186666565632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2969801249651712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179708744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20522993560576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179718735 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33981586 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20747581306880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153864262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚649439085258752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚16459.924104235 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2949278256091136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41760548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 333582997 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 333582997 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0BÈäÒ;N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20522993560576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153864260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚649439103870976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20747671973888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20491345425685 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 _step‚1598 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179708744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 33357+Sξ•!3004 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75741878 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2954596349056647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚670186666565632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2969801249651712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179708744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20522993560576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179718735 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33981586 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20747581306880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153864262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚649439085258752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚16459.924104235 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2949278256091136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41760548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 333582997 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 333582997 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2954595991467285 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75742134 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33981583 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 333573004 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2949278239051776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41760295 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2977065384592896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41861202 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚1601 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚671874760142336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20797665715712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334402351 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20572335089152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180164121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75928293 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651077094426624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2977065384592896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20540950858871 % _timestamp‚1.7842893071736772e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154238228 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2941283668963328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2956493049503744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _runtime‚16499.302386638 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20541291408761 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20572335089152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180154144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚651077113038848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154238230 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34067091 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2941283686002688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360.ÐÊÌN"š;†P M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75928038 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚1601 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚671874760142336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20797665715712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334402351 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20572335089152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180164121 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75928293 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651077094426624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2977065384592896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20540950858871 % _timestamp‚1.7842893071736772e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154238228 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2941283668963328 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2956493049503744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _runtime‚16499.302386638 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20541291408761 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20572335089152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180154144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚651077113038848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154238230 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34067091 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2941283686002688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2956493066543104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20797748001280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154238230 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2977065384592896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41861202 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75931861 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2977213996635136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41863028 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20542302462595 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2956640290533376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2956640290533376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154245860 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20541961912593 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334419049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75932134 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651110541797376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20573689062400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20798773260288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34068836 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34068833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651110523185152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚16499.91001437 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180163196 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2961972499137155 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180173187 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2977213996635136 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2956640307572736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2956640307572736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2977213638249472 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2961972141547793 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671909201584128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41863298 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚671909315057664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180163196 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154245862 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2956640290533376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154245860 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20541961912593 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334419049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75932134 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651110541797376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20573689062400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20798773260288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34068836 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34068833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651110523185152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚16499.91001437 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180163196 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2961972499137155 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180173187 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2977213996635136 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2956640307572736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2956640307572736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2977213638249472 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2961972141547793 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671909201584128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41863298 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚671909315057664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180163196 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154245862 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75931861 ¤çü4ž;/ _wandbruntime‚16499  _runtime‚16499žó gŸ;Š 0loss/FRt2/assignment_loss‚0.7952068987923938 % _timestamp‚1.7842893077877653e+09 _runtime‚16499.913101122 0loss/FRt1/assignment_loss‚1.4611305521690565 3loss/SRqqt1/assignment_loss‚0.31318715037885597 3loss/SRqqt2/assignment_loss‚0.42315488251775796 3loss/FBt1/assignment_loss‚0.0026170069932669768 1loss/SRqqt1/detection_loss‚0.8174140664094564 1loss/SRqqt2/detection_loss‚0.5958340902309334 /loss/FBt1/detection_loss‚0.4161228553024907 %loss/total_loss‚6.799726222211988 /loss/FRt2/detection_loss‚1.1976573656025526 /loss/FBt2/detection_loss‚0.2653137425217973 epoch‚48 trainer/global_step‚19449 _step‚1603 2loss/FBt2/assignment_loss‚0.013770266843875658 /loss/FRt1/detection_loss‚0.4983173444495506MâòF ; 1loss/SRqqt1/detection_loss‚0.8174140664094564 1loss/SRqqt2/detection_loss‚0.5958340902309334 /loss/FBt1/detection_loss‚0.4161228553024907 %loss/total_loss‚6.799726222211988 /loss/FRt2/detection_loss‚1.1976573656025526 /loss/FBt2/detection_loss‚0.2653137425217973 epoch‚48 trainer/global_step‚19449 _step‚1603 2loss/FBt2/assignment_loss‚0.013770266843875658 /loss/FRt1/detection_loss‚0.4983173444495506 0loss/FRt2/assignment_loss‚0.7952068987923938 % _timestamp‚1.7842893077877653e+09 _runtime‚16499.913101122 0loss/FRt1/assignment_loss‚1.4611305521690565 3loss/SRqqt1/assignment_loss‚0.31318715037885597 3loss/SRqqt2/assignment_loss‚0.42315488251775796 3loss/FBt1/assignment_loss‚0.0026170069932669768à ÑJu“¡;‚(:ˆ ›°èÒ™½¢¢ gpu.0.gpu‚88.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ñÔ ¢;‚(:• ›°èÒÃîí£ disk.dm-0.in‚145023.25390625 disk.sda.out‚ 683.1171875 disk.sda3.out‚ 683.1171875$ memory_percent‚97.98238277435303$ disk./.usageGB‚550.9244842529297 disk.drbd2165.in‚0( proc.memory.availableMB‚ 330.56640625! disk.rbd1.out‚124.78173828125# disk.nvme0n1.in‚145023.25390625 disk.rbd0.out‚0 disk.sda2.in‚0 cpu‚25.99839524773295 disk.rbd1.in‚0.0625 disk.rbd3.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚71543.1416015625 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda3.in‚ 0.0078125 disk.sda1.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚71543.1416015625% disk.nvme0n1.out‚71543.1416015625% disk.nvme0c0n1.in‚145023.25390625 disk.dm-1.out‚ 683.1171875# proc.memory.rssMB‚ 2812.66015625* proc.memory.percent‚17.167115211486816 disk.sda.in‚ 0.0078125 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125 network.sent‚ 3.0995071e+07 network.recv‚ 4.350721e+06) disk./.usagePercent‚36.97300439464581 disk.drbd2165.out‚07æç´£;‚(:’ ª°èÒû–½¢ gpu.0.gpu‚88.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚252.389' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚50.47780000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üxàí¤;‚(:” ª°èÒ‚Þठdisk.sda.out‚ 683.359375 disk.sda3.in‚ 0.0078125 disk.sda1.in‚0 disk.drbd2165.out‚0! disk.rbd1.out‚124.84423828125 disk.rbd3.out‚0# disk.dm-0.out‚71553.42529296875' proc.memory.availableMB‚ 330.0390625 network.sent‚ 3.1058169e+07$ disk./.usageGB‚550.9304847717285& disk.nvme0n1.out‚71553.42529296875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0.0625 proc.cpu.threads‚17 disk.sda3.out‚ 683.359375 disk.dm-1.in‚ 0.0078125 disk.dm-0.in‚145023.25390625( disk.nvme0c0n1.out‚71553.42529296875 network.recv‚ 4.355199e+06# disk.nvme0n1.in‚145023.25390625 disk.sda1.out‚0 disk.sda2.in‚0# memory_percent‚97.9856014251709# proc.memory.rssMB‚ 2812.66015625 cpu‚25.99813711635484 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.rbd3.in‚0* proc.memory.percent‚17.167115211486816* disk./.usagePercent‚36.973407094497176 disk.dm-1.out‚ 683.359375% disk.nvme0c0n1.in‚145023.25390625lDúÙ’¥;‚(:‡ ¹°èÒˆ–›¢ gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌM õ#¦;‚(:˜ ¹°èҽ disk.sda3.out‚ 683.7421875 disk.rbd1.in‚0.0625 disk.rbd3.in‚0! disk.rbd1.out‚124.84423828125 disk.sda.out‚ 683.7421875 disk.dm-1.in‚ 0.0078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚71564.77783203125$ disk./.usageGB‚550.9369621276855% disk.nvme0c0n1.in‚145023.25390625 network.recv‚ 4.358625e+06) disk./.usagePercent‚36.97384179529031 disk.sda.in‚ 0.0078125# disk.dm-0.out‚71564.77783203125 disk.sda1.in‚0# proc.memory.rssMB‚ 2812.66015625* proc.memory.percent‚17.167115211486816 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625$ memory_percent‚97.98605442047119( proc.memory.availableMB‚ 329.96484375 network.sent‚ 3.1063207e+07 disk.sda2.in‚0 disk.dm-1.out‚ 683.7421875 disk.dm-0.in‚145023.25390625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125& disk.nvme0n1.out‚71564.77783203125 disk.sda1.out‚0 cpu‚25.99789993915859 proc.cpu.threads‚17ôÊ4§;/ _wandbruntime‚16539  _runtime‚16539:>§Îʨ;Ä & lr-AdamW/pg1‚7.615984771376979e-07 & lr-AdamW/pg2‚7.615984771376979e-07 trainer/global_step‚19499 % _timestamp‚1.7842893471784532e+09 _runtime‚16539.304043832 _step‚1604ì•[iÏ©;É % _timestamp‚1.7842893471784532e+09 _runtime‚16539.304043832 _step‚1604 & lr-AdamW/pg1‚7.615984771376979e-07 & lr-AdamW/pg2‚7.615984771376979e-07 trainer/global_step‚19499Ä ŽF¯«4ª;/ _wandbruntime‚16539  _runtime‚16539¾ZªÛí«;€P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154619830 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2963855118024704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41964316 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2963855100985344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20623030590976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20848923855360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 335238403 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚673597474820608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 $ _timestamp‚1.784289347179498e+09 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2948609202556928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 335228424 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2984478131576320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154619830 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2948609219596288 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2963855100985344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34154338 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚652748532353024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154619828 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚19499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154619828 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76118657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34154341 B`s<ÌDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚652748550965248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2963855118024704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20591567345779 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2984478131576320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚16539.305357492 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2969200769902707 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20591907895669 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180608596 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚652748550965248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2963855118024704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20591567345779 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2984478131576320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚16539.305357492 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2969200769902707 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20591907895669 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180608596 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154619828 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76118657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34154341 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20624384564224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20592578399501 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20624043217920 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76122185 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚652781961111552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154627462 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 $ _timestamp‚1.784289347791891e+09 _step‚1606 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335255101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚652781979723776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180617648 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2948755713228800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180627639 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34156083 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 trainer/global_step‚19499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 335245108 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180627639 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚673631814316032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2964002342014976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚673631919400960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20849939677184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2948755713228800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180627639 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34156083 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 trainer/global_step‚19499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 335245108 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180627639 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚673631814316032 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2964002342014976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚673631919400960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20849939677184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20624384564224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20592578399501 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20624043217920 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76122185 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚652781961111552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154627462 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 $ _timestamp‚1.784289347791891e+09 _step‚1606 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335255101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚652781979723776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180617648 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155001430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675320076937728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20674067431936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654419970279424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2991890878559744 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20673726092800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654419988891648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20673726092800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336064476 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2971217169506304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181063048 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2991891236938240 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181073025 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76308932 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675319967651328 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2976576919983215 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2955934736150528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20899997371904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1609 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2991890878559744¡KuÌ (¾;†P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2955934753189888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2991891236938240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2971217152466944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚16579.072990044 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2976577277572465 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20674067431936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76308666 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20673726092800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336064476 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2971217169506304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181063048 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2991891236938240 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181073025 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76308932 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675319967651328 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2976576919983215 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2955934736150528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20899997371904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _step‚1609 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2991890878559744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155001428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42067341 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20642524382577 trainer/global_step‚19549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155001430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚675320076937728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20674067431936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654419970279424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2991890878559744 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20673726092800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654419988891648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20901010079744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76312491 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2971364410535936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42069158 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34243333 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76312756 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654453417650176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large;¨¼Ãþ_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2976724441708809 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155009062 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42069420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2956081263861760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181072100 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚1610 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155009060 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2992039490601984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654453399037952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2992039490601984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚16579.685145328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336081160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20675080066048 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336091153 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2976724799298171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2956081246822400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675354409117696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20643535436411 trainer/global_step‚19549 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336091153 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155009060 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181082091 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚16579.685145328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336081160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20675080066048 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336091153 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2976724799298171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2956081246822400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675354409117696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20643535436411 trainer/global_step‚19549 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336091153 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155009060 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181082091 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20901010079744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76312491 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2971364410535936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42069158 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34243333 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76312756 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654453417650176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121862144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2976724441708809 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155009062 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42069420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2956081263861760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181072100 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 _step‚1610 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155009060 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2992039490601984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654453399037952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2992039490601984Ê M—4Â;/ _wandbruntime‚16579  _runtime‚16579ÿ)} Ã;‡ 3loss/FBt1/assignment_loss‚0.0033961957237191123 /loss/FRt1/detection_loss‚0.4568742287949308 1loss/SRqqt1/detection_loss‚0.8409971886552999 /loss/FBt2/detection_loss‚0.3228054544661063 trainer/global_step‚19549 -loss/FRt2/detection_loss‚1.29858908670343 0loss/FBt1/detection_loss‚0.29014360886098733 % _timestamp‚1.7842893875623455e+09 _step‚1611 0loss/FRt1/assignment_loss‚1.5136024860631043 0loss/FRt2/assignment_loss‚0.7892273634206504 2loss/FBt2/assignment_loss‚0.008699360820200175 1loss/SRqqt2/detection_loss‚0.5629908473471289 2loss/SRqqt1/assignment_loss‚0.4219473533927546 %loss/total_loss‚6.938479898702282 epoch‚48 _runtime‚16579.687679462 2loss/SRqqt2/assignment_loss‚0.4292067244539711 ¢êÄ;Œ 0loss/FRt1/assignment_loss‚1.5136024860631043 0loss/FRt2/assignment_loss‚0.7892273634206504 2loss/FBt2/assignment_loss‚0.008699360820200175 1loss/SRqqt2/detection_loss‚0.5629908473471289 2loss/SRqqt1/assignment_loss‚0.4219473533927546 %loss/total_loss‚6.938479898702282 epoch‚48 _runtime‚16579.687679462 2loss/SRqqt2/assignment_loss‚0.4292067244539711 3loss/FBt1/assignment_loss‚0.0033961957237191123 /loss/FRt1/detection_loss‚0.4568742287949308 1loss/SRqqt1/detection_loss‚0.8409971886552999 /loss/FBt2/detection_loss‚0.3228054544661063 trainer/global_step‚19549 -loss/FRt2/detection_loss‚1.29858908670343 0loss/FBt1/detection_loss‚0.29014360886098733 % _timestamp‚1.7842893875623455e+09 _step‚1611Ë åÃÉ‹“Å;‚(:ˆ õ°èÒûÂë¢ gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªyO¿Æ;‚(: õ°èÒì©Ñ¥ disk.dm-0.in‚145023.25390625 disk.sda1.in‚0) disk./.usagePercent‚36.97599533066834 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625! disk.dm-0.out‚71614.142578125$ disk.nvme0n1.out‚71614.142578125 disk.drbd2165.out‚0# disk./.usageGB‚550.969051361084 disk.sda3.in‚ 0.0078125! disk.rbd1.out‚124.96923828125* proc.memory.percent‚17.167115211486816 disk.rbd3.in‚0# proc.memory.rssMB‚ 2812.66015625 disk.sda.out‚ 684.6484375# disk.nvme0n1.in‚145023.25390625# proc.memory.availableMB‚330.125 network.sent‚ 3.1153815e+07 network.recv‚ 4.372785e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 684.6484375 cpu‚25.996895359136843 disk.sda3.out‚ 684.6484375& disk.nvme0c0n1.out‚71614.142578125 disk.rbd3.out‚0 disk.drbd2165.in‚0$ memory_percent‚97.98507690429688 proc.cpu.threads‚17 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625Óx#Ç;‚(:’ „±èÒÛžº¢ gpu.0.gpu‚50.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚211.706' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚42.34119999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0føÙ#È;‚(:˜ „±èÒäÄ—¥ disk.rbd0.out‚0 disk.rbd3.in‚0* proc.memory.percent‚17.167115211486816 disk.drbd2165.in‚0 disk.drbd2165.out‚0! disk.rbd1.out‚125.09423828125 disk.rbd3.out‚0 cpu‚25.996659349137033* disk./.usagePercent‚36.976453840162165( disk.nvme0c0n1.out‚71622.13037109375 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 4.377199e+06 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0$ memory_percent‚97.98479080200195 network.sent‚ 3.1203852e+07$ disk./.usageGB‚550.9758834838867 disk.sda.out‚ 685.1640625 disk.sda3.out‚ 685.1640625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 685.1640625 disk.dm-0.in‚145023.25390625# disk.dm-0.out‚71622.13037109375 disk.sda.in‚ 0.0078125 disk.sda2.out‚0& disk.nvme0n1.out‚71622.13037109375& proc.memory.availableMB‚ 330.171875% disk.nvme0c0n1.in‚145023.25390625 disk.sda3.in‚ 0.0078125# proc.memory.rssMB‚ 2812.66015625|`:4É;/ _wandbruntime‚16619  _runtime‚16619uxÉÊ;à & lr-AdamW/pg1‚4.880038362572529e-07 & lr-AdamW/pg2‚4.880038362572529e-07 trainer/global_step‚19599 % _timestamp‚1.7842894268936164e+09 _runtime‚16619.01933804 _step‚16124nÎË;È trainer/global_step‚19599 % _timestamp‚1.7842894268936164e+09 _runtime‚16619.01933804 _step‚1612 & lr-AdamW/pg1‚4.880038362572529e-07 & lr-AdamW/pg2‚4.880038362572529e-07Ì [MGn4Ì;/ _wandbruntime‚16619  _runtime‚16619ZáÌ&Í;P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181527477 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336900528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181527477 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76498969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34328841 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2983953427652973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2999303983921664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚19599 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚656091408205824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2999303983921664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2963260269744128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20692800319595 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155383028 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚656091426818048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2999303625543168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚16619.023370116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42170131 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚677042433797632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155383030 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42170394 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181517500 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155383030 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2999303625543168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2983953070063723 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181517500 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚677042543084032 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7842894268947566e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2978579220987904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34328838 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2978579220987904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42170131 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚677042433797632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155383030 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42170394 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181517500 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155383030 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2999303625543168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2983953070063723 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181517500 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚677042543084032 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 % _timestamp‚1.7842894268947566e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2978579220987904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34328838 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2978579220987904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 336900528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181527477 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76498969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34328841 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2983953427652973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2999303983921664Í ¯_uë4Ï;/ _wandbruntime‚16619  _runtime‚16619¤ ='Ð;‹N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155390660 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181526552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2963406780416000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336917212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842894275061398e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656124836964352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336927205 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181526552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2978726462017536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2978726462017536 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2978726444978176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20725775567872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76503017 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336917212 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155390662 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155390660 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336927205 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚677076993337344 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20693811373317 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42172161 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20725434221568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155390662 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20725434221568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20952051288064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181536543 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20694151923319 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2984100949378679 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76502744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20725775567872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚677076888252416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚656124855576576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1614 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚19599I‡ý'Ñ;N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2984100949378679 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76502744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20725775567872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚677076888252416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚656124855576576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚1614 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚19599 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155390660 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181526552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2963406780416000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336917212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7842894275061398e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656124836964352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336927205 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181526552 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2978726462017536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2978726462017536 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2978726444978176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20725775567872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76503017 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336917212 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155390662 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155390660 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336927205 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚677076993337344 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20693811373317 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42172161 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20725434221568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155390662 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20725434221568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20952051288064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181536543 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20694151923319 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541Î ÔìäM4Ò;/ _wandbruntime‚16619  _runtime‚16619D¤ƒÓ;‰ /loss/FBt1/detection_loss‚0.3438999928766488 %loss/total_loss‚7.003448299341789 epoch‚48 2loss/SRqqt1/assignment_loss‚0.3196834820312501 /loss/FRt1/detection_loss‚0.5129494571301803 0loss/FBt2/detection_loss‚0.36877989054957416 0loss/FRt2/assignment_loss‚0.8112534896593476 /loss/FRt2/detection_loss‚1.1878713823748468 1loss/SRqqt1/detection_loss‚0.9058631500799036 _runtime‚16619.63423231 4loss/FBt1/assignment_loss‚0.00012191798008010458 4loss/FBt2/assignment_loss‚0.00010440820365431591 0loss/SRqqt2/detection_loss‚0.580809367718181 trainer/global_step‚19599 % _timestamp‚1.7842894275088866e+09 _step‚1615 0loss/FRt1/assignment_loss‚1.5095877014928394 1loss/SRqqt2/assignment_loss‚0.462524059245283pƒÍ÷Ô;Ž 1loss/SRqqt1/detection_loss‚0.9058631500799036 _runtime‚16619.63423231 4loss/FBt1/assignment_loss‚0.00012191798008010458 4loss/FBt2/assignment_loss‚0.00010440820365431591 0loss/SRqqt2/detection_loss‚0.580809367718181 trainer/global_step‚19599 % _timestamp‚1.7842894275088866e+09 _step‚1615 0loss/FRt1/assignment_loss‚1.5095877014928394 1loss/SRqqt2/assignment_loss‚0.462524059245283 /loss/FBt1/detection_loss‚0.3438999928766488 %loss/total_loss‚7.003448299341789 epoch‚48 2loss/SRqqt1/assignment_loss‚0.3196834820312501 /loss/FRt1/detection_loss‚0.5129494571301803 0loss/FBt2/detection_loss‚0.36877989054957416 0loss/FRt2/assignment_loss‚0.8112534896593476 /loss/FRt2/detection_loss‚1.1878713823748468Ï \ÖðòÕ;‚(:’ “±èÒćî¢ gpu.0.gpu‚48.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Å]$Ö;‚(:” “±èÒ¼æʤ% disk.nvÚmñûme0c0n1.in‚145023.25390625 disk.rbd3.in‚0# disk.nvme0n1.in‚145023.25390625% disk.nvme0n1.out‚71636.3798828125$ disk./.usageGB‚550.9904518127441 disk.dm-1.out‚ 685.3828125 disk.rbd1.in‚0.0625$ memory_percent‚97.98510074615479 disk.sda2.out‚0 disk.sda3.out‚ 685.3828125 disk.dm-1.in‚ 0.0078125 disk.drbd2165.out‚0( proc.memory.availableMB‚ 330.12109375 disk.sda3.in‚ 0.0078125! disk.rbd1.out‚125.09423828125" disk.dm-0.out‚71636.3798828125 disk.sda1.out‚0* proc.memory.percent‚17.167115211486816 network.sent‚ 3.1208925e+07 network.recv‚ 4.380623e+06) disk./.usagePercent‚36.97743153293543 disk.sda.in‚ 0.0078125 disk.sda2.in‚0 disk.dm-0.in‚145023.25390625 disk.sda1.in‚0 cpu‚25.9964242993515 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 685.3828125' disk.nvme0c0n1.out‚71636.3798828125# proc.memory.rssMB‚ 2812.66015625›ÑËšž×;‚(:“ ¢±èÒé‹­¢ gpu.0.gpu‚94.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚252.395' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.479000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÊØ;‚(:† ¢±èÒŠð´¤ disk.sda.out‚ 685.5546875 disk.rbd3.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625 network.recv‚ 4.384807e+06 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚125.15673828125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚ 0.0078125 disk.sda3.out‚ 685.5546875 disk.sda2.in‚0 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625) disk./.usagePercent‚36.97781631225939$ disk./.usageGB‚550.9961853027344 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625 disk.sda1.out‚0 network.sent‚ 3.1271996e+07 disk.dm-0.in‚145023.25390625 disk.dm-1.out‚ 685.5546875$ memory_percent‚97.99156188964844! disk.nvme0n1.out‚ 71647.171875 disk.drbd2165.in‚0# disk.nvme0c0n1.out‚ 71647.171875# proc.memory.rssMB‚ 2813.64453125 cpu‚25.999650386624737 disk.dm-0.out‚ 71647.171875 disk.drbd2165.out‚0$ proc.memory.availableMB‚329.0625* proc.memory.percent‚17.173123359680176@DXï4Ù;/ _wandbruntime‚16636  _runtime‚16636Œ[”Œ=Ú;†{ =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762619495391846 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8547785878181458 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6228083372116089 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762619495391846 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25158801674842834 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2633051574230194 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7690867185592651 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6250490546226501 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8415139317512512 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953824281692505 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2955244183540344 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6890072822570801 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043658751994371414 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5997698903083801 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27391114830970764 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21119044721126556 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 +jet/accuracy_3_of_4‚0.07847467809915543 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3550480604171753 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022613270208239555 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6543384194374084 _step‚1616 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20643658936023712 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6665154695510864 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6571405529975891 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9906184673309326 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.219415545463562 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.658795177936554 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20436520874500275 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9510270357131958 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5997698903083801 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7"Purity/*FRt1SRqqt1FBt/event_purity‚0.37242391705513 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2393593043088913 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022459974512457848 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6555813550949097 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2261561006307602 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5723271369934082 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037037014961243 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5424817204475403 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7334467172622681 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 $ _timestamp‚1.784289444575039e+09 )loss/val_total_loss‚8.092003966239016 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2715437412261963 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5331695079803467 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6097307205200195 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28648489713668823 6validation_average_jet_accuracy‚0.3713982403278351 trainer/global_step‚19599 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8457856774330139 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25441569089889526 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.592579185962677 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6148545145988464 particle/accuracy_5_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5825087428092957 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6587148904800415 *jet/accuracy_1_of_4‚0.9857255220413208 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5867990255355835 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.776422381401062 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6551250219345093 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947301149368286 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5174999833106995 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1979660540819168 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5724890828132629 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2160274237394333 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7215356230735779 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5473889112472534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22429105639457703 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0893634632229805 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6423335075378418 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7958320379257202 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047128789126873016 *jet/accuracy_1_of_2‚0.7875583171844482 +jet/accuracy_2_of_2‚0.15804046392440796 'particle/f_score‚0.6249242424964905 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6234954595565796 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2501959204673767 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2722392678260803 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 _runtime‚16636.701337094 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23768095672130585 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24891674518585205 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8966571092605591 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26227518916130066 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7193403244018555 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23928315937519073 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6310185790061951 ,jet/accuracy_4_of_4‚0.011098379269242287 particle/accuracy_2_of_2‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3198932409286499 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8021252155303955 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.764668345451355 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5939235091209412 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23805701732635498 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037037014961243 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897313714027405 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02115723118185997 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7286741733551025 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6199995875358582 *jet/accuracy_2_of_3‚0.5738587975502014 +jet/accuracy_3_of_3‚0.05731193348765373 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9156632423400879 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5318278670310974 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35534995794296265 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6136214733123779 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07827772200107574 *jet/accuracy_2_of_4‚0.8052418828010559 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5952374339103699 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875089526176453 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.041441187262535095 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021032636985182762 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6198147535324097 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6232078671455383 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.628603994846344 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7612698674201965 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03585115447640419 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_1‚0.3277672827243805 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7267573475837708 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24390800297260284 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +particle/specificity‚0.9035030603408813 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26227518916130066 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7296673655509949 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2724425792694092 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32954925298690796 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6923924684524536 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4500266909599304 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4885523319244385 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5983161926269531 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2657880187034607 epoch‚48 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *jet/accuracy_1_of_3‚0.9248284697532654 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7278532981872559 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt2FBt/event_purity‚0#$l8Û;‹{ 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5939235091209412 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23805701732635498 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037037014961243 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897313714027405 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02115723118185997 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7286741733551025 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6199995875358582 *jet/accuracy_2_of_3‚0.5738587975502014 +jet/accuracy_3_of_3‚0.05731193348765373 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9156632423400879 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5318278670310974 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35534995794296265 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6136214733123779 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07827772200107574 *jet/accuracy_2_of_4‚0.8052418828010559 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5952374339103699 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875089526176453 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.041441187262535095 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021032636985182762 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6198147535324097 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6232078671455383 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.628603994846344 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7612698674201965 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03585115447640419 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_1‚0.3277672827243805 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7267573475837708 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24390800297260284 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +particle/specificity‚0.9035030603408813 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26227518916130066 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7296673655509949 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2724425792694092 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32954925298690796 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6923924684524536 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4500266909599304 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4885523319244385 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5983161926269531 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2657880187034607 epoch‚48 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *jet/accuracy_1_of_3‚0.9248284697532654 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7278532981872559 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762619495391846 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8547785878181458 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6228083372116089 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762619495391846 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25158801674842834 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2633051574230194 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7690867185592651 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6250490546226501 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8415139317512512 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953824281692505 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2955244183540344 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6890072822570801 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043658751994371414 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5997698903083801 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27391114830970764 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21119044721126556 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 +jet/accuracy_3_of_4‚0.07847467809915543 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3550480604171753 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022613270208239555 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6543384194374084 _step‚1616 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20643658936023712 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6665154695510864 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6571405529975891 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9906184673309326 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.219415545463562 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.658795177936554 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20436520874500275 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9510270357131958 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5997698903083801 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7"Purity/*FRt1SRqqt1FBt/event_purity‚0.37242391705513 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2393593043088913 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022459974512457848 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6555813550949097 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2261561006307602 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5723271369934082 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037037014961243 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5424817204475403 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7334467172622681 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 $ _timestamp‚1.784289444575039e+09 )loss/val_total_loss‚8.092003966239016 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2715437412261963 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5331695079803467 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6097307205200195 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28648489713668823 6validation_average_jet_accuracy‚0.3713982403278351 trainer/global_step‚19599 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8457856774330139 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25441569089889526 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.592579185962677 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6148545145988464 particle/accuracy_5_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5825087428092957 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6587148904800415 *jet/accuracy_1_of_4‚0.9857255220413208 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5867990255355835 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.776422381401062 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6551250219345093 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947301149368286 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5174999833106995 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1979660540819168 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5724890828132629 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2160274237394333 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7215356230735779 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5473889112472534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22429105639457703 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0893634632229805 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6423335075378418 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7958320379257202 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047128789126873016 *jet/accuracy_1_of_2‚0.7875583171844482 +jet/accuracy_2_of_2‚0.15804046392440796 'particle/f_score‚0.6249242424964905 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6234954595565796 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2501959204673767 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2722392678260803 7 Purity/1FRt0SRqqt2²Ýì%FBt/FRt_purity‚0.7278532981872559 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 _runtime‚16636.701337094 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23768095672130585 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24891674518585205 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8966571092605591 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26227518916130066 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7193403244018555 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23928315937519073 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6310185790061951 ,jet/accuracy_4_of_4‚0.011098379269242287 particle/accuracy_2_of_2‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3198932409286499 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8021252155303955 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.764668345451355 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255Ð —ºÏ÷œÜ;‚(:‘ ±±èÒ‚¹£ gpu.0.gpu‚6.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šw§·Ý;‚(:“ ±±èÒú¹”¥ disk.sda.out‚ 685.90625 network.recv‚ 4.388312e+06* disk./.usagePercent‚36.978298118438836' proc.memory.availableMB‚ 329.1015625 cpu‚26.000007650288993 disk.dm-1.out‚ 685.90625# disk.nvme0n1.in‚145023.25390625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125 disk.dm-0.in‚145023.25390625# disk.dm-0.out‚71659.89404296875% disk.nvme0c0n1.in‚145023.25390625* proc.memory.percent‚17.173123359680176 disk.sda3.in‚ 0.0078125 disk.sda1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2813.64453125 network.sent‚ 3.1277084e+07 disk.rbd1.in‚0.0625 disk.rbd3.in‚0! disk.rbd1.out‚125.15673828125 disk.rbd3.out‚0& disk.nvme0n1.out‚71659.89404296875 disk.sda.in‚ 0.0078125$ memory_percent‚97.99132347106934 proc.cpu.threads‚17$ disk./.usageGB‚551.0033645629883 disk.sda3.out‚ 685.90625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚71659.89404296875uª’Þ;‚(:‡ À±èÒ«üˆ¤ gpu.0.gpu‚23.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mˆ&ß;‚(:› À±èÒƒ÷æ¤ disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 686.07421875 cpu‚25.999754479285944 disk.rbd3.in‚0 disk.sda1.in‚0$ disk./.usageGB‚551.0090179443359 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚97.99134731292725( proc.memory.availableMB‚ 329.09765625* proc.memory.percent‚17.173123359680176& disk.nvme0n1.out‚71668.77294921875 disk.dm-1.out‚ 686.07421875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0.0625# disk.nvme0n1.in‚145023.25390625 network.sent‚ 3.1334594e+07 network.recv‚ 4.39264e+06) disk./.usagePercent‚36.97867752160458 disk.rbd3.out‚0# disk.dm-0.out‚71668.77294921875# proc.memory.rssMB‚ 2813.64453125 proc.cpu.threads‚17 disk.dm-1.in‚ 0.0078125 disk.dm-0.in‚145023.25390625 disk.rbd0.out‚0 disk.sda.out‚ 686.07421875 disk.sda2.in‚0! disk.rbd1.out‚125.21923828125 disk.drbd2165.out‚0% disk.nvme0c0n1.in‚145023.25390625( disk.nvme0c0n1.out‚71668.77294921875 disk.sda.in‚ 0.0078125`õÄ4à;/ _wandbruntime‚16676  _runtime‚16676\ÍCLÌá;Æ ' lr-AdamW/pg1‚3.7392429753104294e-07 ' lr-AdamW/pg2‚3.7392429753104294e-07 trainer/global_step‚19649 % _timestamp‚1.7842894839067163e+09 _runtime‚16676.032341025 _step‚1617Ipû Ñâ;Ë ' lr-AdamW/pg1‚3.7392429753104294e-07 ' lr-AdamW/pg2‚3.7392429753104294e-07 trainer/global_step‚19649 % _timestamp‚1.7842894839067163e+09 _runtime‚16676.032341025 _step‚1617Ñ órÇA4ã;/ _wandbruntime‚16676  _runtime‚16676Î Šw(ä;€P XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76722711 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337927568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚657778776844288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842894839078112e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20793191641088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2987870822080512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42303845 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21020474228224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20761477081622 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2987870839119872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337937547 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182042274 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155885296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21020394039808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚16676.033599922 _step‚1618 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34418608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20792850301952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2987870839119872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3008664030760960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2972494483591168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182052251 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20761136531732 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3008663672382464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3008663672382464 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20793191641088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182042274 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182052251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76722453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155885294 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3008664030760960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20792850301952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34418611 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2993255603083540 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155885294 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2993255960672790 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42304100 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678799251072512 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2972494483591168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182052251 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20761136531732 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3008663672382464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3008663672382464 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20793191641088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182042274 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182052251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76722453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155885294 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3008664030760960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20792850301952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34418611 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2993255603083540 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155885294 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2993255960672790 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42304100 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678799251072512 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76726324 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337954245 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2972640994263040 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337954245 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155892926 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678833714705408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42305971 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182051326 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3008811926039040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678833620106240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2988018063110144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2988018080149504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20793862928896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2993403124809134 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2988018080149504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20794204275200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657812205602816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657812186990592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337944252 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚16676.645415493 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3008812284424704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34420356 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337944252 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20793862928896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21021509102592 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182061317 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2988018063110144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21021433115648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155892926 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155892928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154;ˆ8'è;N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21021433115648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155892926 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155892928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155892928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34420353 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76726324 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337954245 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2972640994263040 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337954245 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155892926 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678833714705408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42305971 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182051326 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3008811926039040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678833620106240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2988018063110144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2988018080149504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20793862928896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2993403124809134 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2988018080149504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20794204275200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚657812205602816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚657812186990592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337944252 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚16676.645415493 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3008812284424704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34420356 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337944252 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20793862928896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21021509102592 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182061317 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2988018063110144Ó I‰é4é;/ _wandbruntime‚16676  _runtime‚166762ãö+ê;‰ %loss/total_loss‚6.746154750605202 1loss/SRqqt1/detection_loss‚0.6819129134056656 1loss/SRqqt2/detection_loss‚0.5689521425571104 /loss/FBt2/detection_loss‚0.5496432252810337 trainer/global_step‚19649 epoch‚49 3loss/FBt1/assignment_loss‚0.0067891481762466215 /loss/FRt2/detection_loss‚1.1217804446351762 0loss/FBt1/detection_loss‚0.35559601178306877 % _timestamp‚1.7842894845220706e+09 _runtime‚16676.647467266 0loss/FRt1/assignment_loss‚1.4345272515906768 2loss/FBt2/assignment_loss‚0.022299328397918927 _step‚1620 0loss/FRt2/assignment_loss‚0.7564234898146556 2loss/SRqqt1/assignment_loss‚0.3102008796359716 2loss/SRqqt2/assignment_loss‚0.4148847660792384 /loss/FRt1/detection_loss‚0.5231451492484392g˜së;Ž epoch‚49 3loss/FBt1/assignment_loss‚0.0067891481762466215 /loss/FRt2/detection_loss‚1.1217804446351762 0loss/FBt1/detection_loss‚0.35559601178306877 % _timestamp‚1.7842894845220706e+09 _runtime‚16676.647467266 0loss/FRt1/assignment_loss‚1.4345272515906768 2loss/FBt2/assignment_loss‚0.022299328397918927 _step‚1620 0loss/FRt2/assignment_loss‚0.7564234898146556 2loss/SRqqt1/assignment_loss‚0.3102008796359716 2loss/SRqqt2/assignment_loss‚0.4148847660792384 /loss/FRt1/detection_loss‚0.5231451492484392 %loss/total_loss‚6.746154750605202 1loss/SRqqt1/detection_loss‚0.6819129134056656 1loss/SRqqt2/detection_loss‚0.5689521425571104 /loss/FBt2/detection_loss‚0.5496432252810337 trainer/global_step‚19649Ô é–©Kì;‚(:’ ϱèÒ€’ठgpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.405' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.08099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w,Sí;‚(:Ž ϱèÒ¶²œ¦ proc.cpu.threads‚17' disk.nvme0c0n1.out‚71681.2490234375 disk.sda.in‚ 0.0078125 disk.sda.out‚ 686.34375 disk.sda3.out‚ 686.34375 disk.sda1.out‚0( proc.memory.availableMB‚ 329.11328125 network.recv‚ 4.39613e+06 disk.rbd1.in‚0.0625! disk.rbd1.out‚125.21923828125% disk.nvme0n1.out‚71681.2490234375 disk.sda1.in‚0 cpu‚25.999510941920253 disk.rbd0.in‚0 disk.sda3.in‚ 0.0078125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0078125) disk./.usagePercent‚36.97905692477032 disk.sda2.in‚0 disk.dm-1.out‚ 686.34375 disk.dm-0.in‚145023.25390625# disk.nvme0n1.in‚145023.25390625# proc.memory.rssMB‚ 2813.64453125$ disk./.usageGB‚551.0146713256836 disk.drbd2165.out‚0# memory_percent‚97.9912519454956 network.sent‚ 3.1352774e+07 disk.rbd0.out‚0% disk.nvme0c0n1.in‚145023.25390625" disk.dm-0.out‚71681.2490234375 disk.drbd2165.in‚0* proc.memory.percent‚17.1731233596801760ŽX¾“î;‚(:ˆ Þ±èÒ™¸·¢ gpu.0.gpu‚89.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚253.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýªàEï;‚(:ƒ Þ±èÒž­Â¤ disk.rbd0.out‚0 disk.nvme0c0n1.out‚ 71691.125 disk.sda.out‚ 686.50390625 disk.rbd3.in‚0# disk.nvme0n1.in‚145023.25390625 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625 disk.dm-1.in‚ 0.0078125' proc.memory.availableMB‚ 328.8359375 network.sent‚ 3.1402946e+07 disk.rbd1.in‚0.0625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 686.50390625* disk./.usagePercent‚36.979475497088806$ disk./.usageGB‚551.0209083557129 disk.dm-1.out‚ 686.50390625 disk.dm-0.in‚145023.25390625 disk.drbd2165.in‚0$ memory_percent‚97.99294471740723# proc.memory.rssMB‚ 2813.64453125 proc.cpu.threads‚17 network.recv‚ 4.400933e+06 disk.sda2.out‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 71691.125 disk.sda.in‚ 0.0078125 disk.sda2.in‚0! disk.rbd1.out‚125.28173828125 disk.drbd2165.out‚0 disk.nvme0n1.out‚ 71691.125* proc.memory.percent‚17.173123359680176 cpu‚25.99923315263696Ä{“ð;‚(:ˆ í±è҈믢 gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚267.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 U‡'ñ;‚(:‘ í±èÒ¨—Ó¥ disk.rbd0.out‚0 disk.sda.out‚ 686.89453125 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 686.89453125 disk.dm-1.out‚ 686.89453125$ memory_percent‚97.99294471740723$ disk./.usageGB‚551.0265655517578% disk.nvme0c0n1.in‚145023.25390625 disk.dm-1.in‚ 0.0078125 disk.rbd1.in‚0.0625! disk.rbd1.out‚125.28173828125 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2813.64453125) disk./.usagePercent‚36.97985515626208 disk.rbd3.in‚0' proc.memory.availableMB‚ 328.8359375 disk.sda2.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.173123359680176# disk.nvme0n1.out‚71703.30078125 disk.rbd0.in‚0 disk.dm-0.in‚145023.25390625 disk.dm-0.out‚71703.30078125 proc.cpu.threads‚17 network.sent‚ 3.1407985e+07 disk.sda.in‚ 0.0078125 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625% disk.nvme0c0n1.out‚71703.30078125 disk.drbd2165.in‚0 cpu‚25.99897654480685 network.recv‚ 4.404444e+06 disk.sda2.in‚0'.Â4ò;/ _wandbruntime‚16715  _runtime‚16715§X$¿Êó;Ä $ _timestamp‚1.784289523757403e+09 _runtime‚16715.88299402 _step‚1621 ' lr-AdamW/pg1‚2.7500004040989115e-07 ' lr-AdamW/pg2‚2.7500004040989115e-07 trainer/global_step‚19699£…5~Ïô;É _runtime‚16715.88299402 _step‚1621 ' lr-AdamW/pg1‚2.7500004040989115e-07 ' lr-AdamW/pg2‚2.7500004040989115e-07 trainer/global_step‚19699 $ _timestamp‚1.784289523757403e+09Õ þ+Ù4õ;/ _wandbruntime‚16715  _runtime‚16715×Ùé(ö;€P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚680521985998848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2995232873562112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7842895237584581e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3016076419365888 _runtime‚16715.88432314 _step‚1622 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182506703 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2995232890601472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21071878417408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3016076419365888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156266896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21071789840384 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20811753018640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20843887142912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338773599 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚19699 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚680522093188096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚659450196158464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2995232873562112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3000632110753298 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3016076419365888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_p]Ü¢RGool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156266896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21071789840384 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20811753018640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20843887142912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338773599 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚19699 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚680522093188096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚659450196158464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2995232873562112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3000632110753298 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3016076419365888 _runtime‚16715.88432314 _step‚1622 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182506703 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2995232890601472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21071878417408 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2979966510817280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2995380114591744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338790297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42410584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76918187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21072900243968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76918468 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659483624916992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182515769 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42410862 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3000779274889642 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2979966527856640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156274526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7842895243695617e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338790297 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338780304 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3016225031408128 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚19699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2995380131631104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34507603 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680556438688256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3016224673022464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚659483643529216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21072813771264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20844899777024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156274528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156274526 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182515769 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20813104622364 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3016225031408128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088éÛÈ'ú;N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34507603 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚680556438688256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3016224673022464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚659483643529216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21072813771264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20844899777024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156274528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156274526 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182515769 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20813104622364 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3016225031408128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1623 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2995380131631104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3000779632479004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2979966510817280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2995380114591744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338790297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42410584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76918187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21072900243968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76918468 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659483624916992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182515769 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42410862 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3000779274889642 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2979966527856640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156274526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7842895243695617e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338790297 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338780304 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3016225031408128 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚19699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2995380131631104× 0k 4û;/ _wandbruntime‚16716  _runtime‚16716,SÏ°ü;Š 0loss/FRt2/assignment_loss‚0.7319732396961092 3loss/SRqqt2/assignment_loss‚0.41806587551280977 .loss/FRt2/detection_loss‚1.150984443198404 % _timestamp‚1.7842895243724086e+09 2loss/SRqqt1/assignment_loss‚0.3150206568999419 3loss/FBt2/assignment_loss‚6.139039536025491e-06 1loss/SRqqt1/detection_loss‚0.8657964745652862 _runtime‚16716.497755308 _step‚1624 0loss/FRt1/assignment_loss‚1.4835075742545798 /loss/FRt1/detection_loss‚0.5059393582774179 0loss/FBt2/detection_loss‚0.41882981621188264 %loss/total_loss‚6.831209808952417 epoch‚49 trainer/global_step‚19699 3loss/FBt1/assignment_loss‚7.849743414248226e-05 1loss/SRqqt2/detection_loss‚0.5507501986257257 /loss/FBt1/detection_loss‚0.3902575352365823žËå€ý; _runtime‚16716.497755308 _step‚1624 0loss/FRt1/assignment_loss‚1.4835075742545798 /loss/FRt1/detection_loss‚0.5059393582774179 0loss/FBt2/detection_loss‚0.41882981621188264 %loss/total_loss‚6.831209808952417 epoch‚49 trainer/global_step‚19699 3loss/FBt1/assignment_loss‚7.849743414248226e-05 1loss/SRqqt2/detection_loss‚0.5507501986257257 /loss/FBt1/detection_loss‚0.3902575352365823 0loss/FRt2/assignment_loss‚0.7319732396961092 3loss/SRqqt2/assignment_loss‚0.41806587551280977 .loss/FRt2/detection_loss‚1.150984443198404 % _timestamp‚1.7842895243724086e+09 2loss/SRqqt1/assignment_loss‚0.3150206568999419 3loss/FBt2/assignment_loss‚6.139039536025491e-06 1loss/SRqqt1/detection_loss‚0.8657964745652862Ø 9K(§žþ;‚(:“ ü±èÒÏ gpu.0.gpu‚70.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ˜m¼ÿ;‚(: ü±èÒœè¥ disk.rbd3.out‚0$ disk.nvme0n1.out‚71725.388671875 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚71725.388671875 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚125.34423828125) disk./.usagePercent‚36.98055303280042 disk.rbd0.out‚0 disk.sda3.in‚ 0.0078125 network.sent‚ 3.1413049e+07 disk.dm-1.out‚ 688.859375 disk.rbd3.in‚0# proc.memory.rssMB‚ 2813.64453125 network.recv‚ 4.407962e+06 disk.rbd0.in‚0 disk.sda3.out‚ 688.859375 disk.rbd1.in‚0.0625* proc.memory.percent‚17.173123359680176 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0! disk.dm-0.out‚71725.388671875$ disk./.usageGB‚551.0369644165039# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 688.859375 disk.dm-0.in‚145023.25390625$ memory_percent‚97.99292087554932( proc.memory.availableMB‚ 328.83984375% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.sda2.out‚0 cpu‚25.99875335159074yÇƺ€<‚(: ‹²èÒÉÛ§¤$ disk./.usageGB‚551.0398635864258! disk.rbd1.out‚125.34423828125 disk.rbd3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2813.64453125* proc.memory.percent‚17.173123359680176# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0 disk.sda.in‚ 0.0078125 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 cpu‚25.998476316880883% disk.nvme0c0n1.out‚71744.21484375 disk.dm-1.out‚ 691.18359375# memory_percent‚97.9914665222168& proc.memory.availableMB‚ 329.078125 network.sent‚ 3.1476109e+07 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 691.18359375) disk./.usagePercent‚36.98074759852645% disk.nvme0c0n1.in‚145023.25390625 disk.sda.out‚ 691.18359375 disk.rbd0.in‚0 disk.dm-1.in‚ 0.0078125 disk.sda1.out‚0 disk.rbd1.in‚0.0625 disk.rbd3.out‚0 disk.dm-0.out‚71744.21484375 disk.dm-0.in‚145023.25390625# disk.nvme0n1.out‚71744.21484375 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.recv‚ 4.412575e+06͈—´“<‚(:ˆ ‹²èÒÇõð£ gpu.0.gpu‚80.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&×Øó“‚<‚(:ˆ š²èÒÞ’Ó¢ gpu.0.gpu‚92.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úEë`±ƒ<‚(:– š²èÒÚ¯ˆ¥ disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚125.46923828125* proc.memory.percent‚17.173123359680176 network.sent‚ 3.1481193e+07% disk.nvme0c0n1.in‚145023.25390625 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 691.609375 disk.rbd3.in‚0$ memory_percent‚97.99168109893799# proc.memory.rssMB‚ 2813.64453125 disk.drbd2165.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚71756.39306640625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.in‚145023.25390625# disk.dm-0.out‚71756.39306640625 disk.drbd2165.out‚0 disk.sda.out‚ 691.609375 disk.rbd~Ì p1.in‚0.0625 disk.rbd3.out‚0 disk.sda3.out‚ 691.609375 cpu‚25.998237126218495$ disk./.usageGB‚551.0464019775391 proc.cpu.threads‚17# disk.nvme0n1.in‚145023.25390625& disk.nvme0n1.out‚71756.39306640625 disk.sda2.in‚0( proc.memory.availableMB‚ 329.04296875 network.recv‚ 4.415594e+06) disk./.usagePercent‚36.98118639544013>ah4„</ _wandbruntime‚16755  _runtime‚16755fNÆ,Ê…<Ä ' lr-AdamW/pg2‚1.9123741861399355e-07 trainer/global_step‚19749 # _timestamp‚1.78428956368723e+09 _runtime‚16755.812849372 _step‚1625 ' lr-AdamW/pg1‚1.9123741861399355e-07èðφ<É # _timestamp‚1.78428956368723e+09 _runtime‚16755.812849372 _step‚1625 ' lr-AdamW/pg1‚1.9123741861399355e-07 ' lr-AdamW/pg2‚1.9123741861399355e-07 trainer/global_step‚19749Ù Ê Sí4‡</ _wandbruntime‚16755  _runtime‚16755iÛ9ž(ˆ<€P @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156648494 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34593108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚682244837266944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156648496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3023489524727808 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20862369505548 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20894241305600 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3002594942083072 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156648496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77105949 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20862710055438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚19749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339599672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42512562 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661121652697088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77105670 _runtime‚16755.817380626 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182951178 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34593111 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42512838 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21123184569856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 $ _timestamp‚1.784289563688288e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20894241305600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3008008260833806 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2987145533739008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3002594925043712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3023489524727808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _step‚1626 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3023489166349312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3023489166349312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156648494 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339609651 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20894582644736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20894582644736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21123091798528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182961155 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3008007903244556 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚661121634084864 j (‰<…P WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3023489166349312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3023489166349312 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156648494 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339609651 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20894582644736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20894582644736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21123091798528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182961155 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3008007903244556 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚661121634084864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156648494 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34593108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚682244837266944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156648496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3023489524727808 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20862369505548 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20894241305600 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3002594942083072 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156648496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77105949 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20862710055438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚19749 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339599672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42512562 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661121652697088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77105670 _runtime‚16755.817380626 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182951178 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34593111 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42512838 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21123184569856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 $ _timestamp‚1.784289563688288e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20894241305600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3008008260833806 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2987145533739008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3002594925043712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3023489524727808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _step‚1626 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Ú l!¼4Š</ _wandbruntime‚16756  _runtime‚16756FðÎ'‹<ŠN KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3002742183112704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182970221 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182960230 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2987292044410880 % _timestamp‚1.7842895642994585e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2987292061450240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚19749 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156656128 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3023637778391552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339616356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20863380559270 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156656128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚682279282205696 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77109431 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3023637420005888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156656126 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚16756.42531066 _step‚1627 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682279170829312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661155081455616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21124200750080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3002742166073344 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339626349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339626349 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3002742166073344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚661155062843392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20863721109272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34594856 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156656126 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182970221 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3023637778391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42514843 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77109699 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21124107985920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20895595278848 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2987292061450240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚19749 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156656128 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3023637778391552 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339616356 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20863380559270 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156656128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚682279282205696 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77109431 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3023637420005888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156656126 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚16756.42531066 _step‚1627 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682279170829312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚661155081455616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21124200750080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3002742166073344 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339626349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339626349 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3002742166073344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚661155062843392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20863721109272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34594856 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156656126 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182970221 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3023637778391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42514843 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77109699 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21124107985920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20895595278848 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3030901913332736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157030094 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3030902271711232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20945278146560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20945278146560 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21174378380800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42616724 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚19799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3009956993564672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42616983 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183415607 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2994471067332608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚16795.797499229 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183405630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20912985992456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20944936807424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77297082 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34680361 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3009956976525312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20913326542346 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157030094 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088¾Îëz½!™<†P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20945278146560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20945278146560 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21174378380800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42616724 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚19799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3009956993564672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42616983 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183415607 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2994471067332608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚16795.797499229 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183405630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20912985992456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20944936807424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77297082 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34680361 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3009956976525312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20913326542346 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157030094 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3030902271711232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚1630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3030901913332736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157030094 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3030902271711232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872Þ ä—‚à4š</ _wandbruntime‚16796  _runtime‚16796…QT3'›<ŠN KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3015531575050658 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20946290780672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚684001890732032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34682106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340462401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157037726 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77300907 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3010104217554944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157037728 _runtime‚16796.406936004 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚684002004205568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚662826500769792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3010104234594304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3031050525374976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3010104217554944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21175389962240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20914337596180 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340452408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2994617578004480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3031050525374976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77301192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚662826519382016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21175484823552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20946290780672 trainer/global_step‚19799 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34682103y ":'œ<N EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340462401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157037726 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77300907 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3010104217554944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157037728 _runtime‚16796.406936004 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚684002004205568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚662826500769792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3010104234594304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3031050525374976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3010104217554944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21175389962240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20914337596180 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340452408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2994617578004480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3031050525374976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77301192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚662826519382016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21175484823552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20946290780672 trainer/global_step‚19799 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34682103 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3015531575050658 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20946290780672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚684001890732032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34682106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438ß ÞG©û4</ _wandbruntime‚16796  _runtime‚16796,»ÇÅž<‹ epoch‚49 0loss/FRt1/assignment_loss‚1.4584627658191216 3loss/SRqqt2/assignment_loss‚0.40775993563407015 /loss/FBt2/detection_loss‚0.3278772032177325 trainer/global_step‚19799 % _timestamp‚1.7842896042838655e+09 _runtime‚16796.409192264 _step‚1632 0loss/FRt2/assignment_loss‚0.6767026924862052 2loss/SRqqt1/assignment_loss‚0.3866400486024526 3loss/FBt1/assignment_loss‚0.0013283227936584255 2loss/FBt2/assignment_loss‚0.004588255280966942 0loss/FRt1/detection_loss‚0.44131115206352206 1loss/SRqqt2/detection_loss‚0.5531606324877004 0loss/FBt1/detection_loss‚0.37152418783559926 /loss/FRt2/detection_loss‚1.2279318831112556 1loss/SRqqt1/detection_loss‚0.8024143090877726 %loss/total_loss‚6.659701388420057´„zŸ< 1loss/SRqqt2/detection_loss‚0.5531606324877004 0loss/FBt1/detection_loss‚0.37152418783559926 /loss/FRt2/detection_loss‚1.2279318831112556 1loss/SRqqt1/detection_loss‚0.8024143090877726 %loss/total_loss‚6.659701388420057 epoch‚49 0loss/FRt1/assignment_loss‚1.4584627658191216 3loss/SRqqt2/assignment_loss‚0.40775993563407015 /loss/FBt2/detection_loss‚0.3278772032177325 trainer/global_step‚19799 % _timestamp‚1.7842896042838655e+09 _runtime‚16796.409192264 _step‚1632 0loss/FRt2/assignment_loss‚0.6767026924862052 2loss/SRqqt1/assignment_loss‚0.3866400486024526 3loss/FBt1/assignment_loss‚0.0013283227936584255 2loss/FBt2/assignment_loss‚0.004588255280966942 0loss/FRt1/detection_loss‚0.44131115206352206à Y÷kÙž <‚(:“ DzèÒêóð¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.515800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s‹¡<‚(:‹ DzèÒìõ‰¥% disk.nvme0c0n1.in‚145023.25390625 disk.sda.out‚695.8125 disk.rbd3.in‚0 disk.drbd2165.in‚0& proc.memory.availableMB‚ 328.796875 proc.cpu.threads‚17 network.recv‚ 4.426469e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0078125 disk.dm-0.in‚145023.25390625" disk.dm-0.out‚71814.2138671875% disk.nvme0n1.out‚71814.2138671875$ disk./.usageGB‚551.0697212219238 disk.rbd3.out‚0 disk.sda3.out‚695.8125 disk.dm-1.in‚ 0.0078125! disk.rbd1.out‚125.53173828125 disk.sda1.in‚0) disk./.usagePercent‚36.98275136949694 disk.rbd1.in‚0.0625$ memory_percent‚97.99318313598633# proc.memory.rssMB‚ 2813.64453125* proc.memory.percent‚17.173123359680176 disk.sda2.in‚0 disk.dm-1.out‚695.8125 disk.rbd0.in‚0 disk.sda.in‚ 0.0078125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚71814.2138671875# disk.nvme0n1.in‚145023.25390625 disk.sda1.out‚0 cpu‚25.997499757597147 network.sent‚ 3.1554473e+07/Aû¼“¢<‚(:ˆ Ö²èÒÌìÈ¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G_…&£<‚(:› Ö²èÒƒ·³¤ disk.sda.in‚ 0.0078125 disk.sda2.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 328.83984375# proc.memory.rssMB‚ 2813.64453125 disk.drbd2165.out‚0 disk.sda3.out‚ 699.87109375 cpu‚25.99725209960206 disk.rbd0.in‚0! disk.rbd1.out‚125.59423828125$ disk./.usageGB‚551.0740966796875 disk.rbd0.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.173123359680176 disk.sda1.in‚0( disk.nvme0c0n1.out‚71830.05224609375 disk.dm-1.out‚ 699.87109375 disk.rbd1.in‚0.0625# disk.dm-0.out‚71830.05224609375 proc.cpu.threads‚17 network.sent‚ 3.1617534e+07 disk.sda1.out‚0% disk.nvme0c0n1.in‚145023.25390625 disk.rbd3.in‚0 network.recv‚ 4.430791e+06# disk.nvme0n1.in‚145023.25390625 disk.drbd2165.in‚0$ memory_percent‚97.99292087554932) disk./.usagePercent‚36.98304501013872& disk.nvme0n1.out‚71830.05224609375 disk.sda.out‚ 699.87109375 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.dm-0.in‚145023.25390625âZ®bž¤<‚(:“ å²èÒ½¦´¢ gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z3ü &¥<‚(:› å²èÒ˜¼÷¤ disk.sda2.in‚0 disk.sda3.in‚ 0.0078125! disk.rbd1.out‚125.59423828125* proc.memory.percent‚17.173123359680176 cpu‚25.9970306146114# disk.nvme0n1.in‚145023.25390625 disk.rbd0.out‚0 disk.dm-1.in‚ 0.0078125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚71852.23974609375 network.recv‚ 4.434469e+06 disk.sda.in‚ 0.0078125 disk.sda.out‚ 702.91796875 disk.rbd1.in‚0.0625$ disk./.usageGB‚551.0791511535645& disk.nvme0n1.out‚71852.23974609375 disk.drbd2165.out‚0# disk.dm-0.out‚71852.23974609375 disk.drbd2165.in‚0 disk.sda3.out‚ 702.91796875 disk.dm-0.in‚145023.25390625$ memory_percent‚97.99439907073975 network.sent‚ 3.1622572e+07% disk.nvme0c0n1.in‚145023.25390625 disk.sda2.out‚0 disk.dm-1.out‚ 702.91796875( proc.memory.availableMB‚ 328.59765625# proc.memory.rssMB‚ 2813.64453125 proc.cpu.threads‚17* disk./.usagePercent‚36.983384220121586 disk.sda1.in‚0œrU4¦</ _wandbruntime‚16835  _runtime‚16835§(Vȧ< _runtime‚16835.54946285 _step‚1633 & lr-AdamW/pg1‚6.921762651559872e-08 & lr-AdamW/pg2‚6.921762651559872e-08 trainer/global_step‚19849 $ _timestamp‚1.784289643423798e+09a_ ´Í¨<Ç $ _timestamp‚1.784289643423798e+09 _runtime‚16835.54946285 _step‚1633 & lr-AdamW/pg1‚6.921762651559872e-08 & lr-AdamW/pg2‚6.921762651559872e-08 trainer/global_step‚19849á ûÏ©4©</ _wandbruntime‚16835  _runtime‚16835î÷Ë©Qª<P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183860082 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20995973648384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341281755 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34767611 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 % _timestamp‚1.7842896434248827e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157411694 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157411694 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3017319045046272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20995632309248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚19849 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42720208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183870059 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3038315018694656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3017319028006912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685690114156544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21225604218880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77487816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42720499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34767608 _runtime‚16835.550666524 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3022760203405572 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20995632309248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚664464509937664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3001796600926208 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183860082 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3022760560994822 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20963602479364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77488110 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20963943029254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341271776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3017319028006912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3001796617965568 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157411696 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3038314660316160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3038314660316160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21225703281664 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183870059 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3038315018694656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3017319028006912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685690114156544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21225604218880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77487816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42720499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34767608 _runtime‚16835.550666524 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3022760203405572 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20995632309248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚664464509937664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3001796600926208 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183860082 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3022760560994822 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20963602479364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77488110 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20963943029254 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341271776 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3017319028006912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3001796617965568 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157411696 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3038314660316160 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3038314660316160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21225703281664 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341298453 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183879125 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157419328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3017466269036544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77491572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 _step‚1635 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21226634848768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 341298453 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341288460 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157419326 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚16836.162394295 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20996986282496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157419326 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183879125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34769356 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3022907725131166 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20996644936192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42722485 trainer/global_step‚19849 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3038463272358400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20996644936192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚16836.162394295 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20996986282496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157419326 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183879125 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34769356 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3022907725131166 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20996644936192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42722485 trainer/global_step‚19849 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3038463272358400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20996644936192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341298453 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183879125 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157419328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3017466269036544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77491572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 _step‚1635 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21226634848768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 341298453 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341288460 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157419326 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3045727407299584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157793294 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21276994026496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157793294 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3045727765678080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77678496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42823635 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3024681079488512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42823380 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666135966476288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184314534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342107828 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34854858 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21046669150208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842896833562636e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3024681079488512 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157793296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚1638 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3045727407299584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚16875.482033615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3009122134519808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157793296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3024681096527872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚666135947864064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184324511 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3024681096527872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21014559516162 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3045727765678080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34854861 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184324511 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3030136711075330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚687412857507840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3030136353486080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21014218966272 trainer/global_step‚19899?× (½<†P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34854858 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21046669150208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842896833562636e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3024681079488512 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157793296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _step‚1638 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3045727407299584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚16875.482033615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3009122134519808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157793296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3024681096527872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚666135947864064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184324511 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3024681096527872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21014559516162 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3045727765678080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34854861 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184324511 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3030136711075330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚687412857507840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3030136353486080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21014218966272 trainer/global_step‚19899 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21046669150208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77678238 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3045727407299584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157793294 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21276994026496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157793294 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3045727765678080 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77678496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42823635 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3024681079488512 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42823380 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666135966476288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184314534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342107828 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979æ g›}4¾</ _wandbruntime‚16876  _runtime‚16876l‚Ï|¿<‹N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342124512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3030284232801036 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34856606 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3024828320518144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184333577 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚687447414913536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21015230019994 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184333577 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3045876019341824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157800928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocatí»ôr•ion.large_pool.allocated‚ 157800928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21278019678720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚16876.094909345 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342134505 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3045875660956160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77682050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚687447307731456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1639 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34856603 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184323586 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666169395234816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚19899 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77682311 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 % _timestamp‚1.7842896839690788e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342124512 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157800926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42825705 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3045876019341824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21047681784320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3009268645191680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21047681784320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3024828337557504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚666169376622592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21047340438016 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342134505 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184333577 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3045876019341824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157800928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157800928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21278019678720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚16876.094909345 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342134505 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3045875660956160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77682050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚687447307731456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1639 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34856603 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184323586 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666169395234816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚19899 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77682311 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 % _timestamp‚1.7842896839690788e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342124512 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157800926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42825705 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3045876019341824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21047681784320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3009268645191680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21047681784320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362ç aÍ»è4Á</ _wandbruntime‚16876  _runtime‚16876kY*Â<Š -loss/FRt2/detection_loss‚1.21111033478072 _step‚1640 2loss/SRqqt2/assignment_loss‚0.4190470538612506 4loss/FBt2/assignment_loss‚1.2500137249560338e-05 /loss/FRt1/detection_loss‚0.5189044265889576 1loss/SRqqt1/detection_loss‚0.7621442069042536 epoch‚49 % _timestamp‚1.7842896839709752e+09 3loss/FBt1/assignment_loss‚0.0007394015392905231 1loss/SRqqt2/detection_loss‚0.5931135278097645 /loss/FBt2/detection_loss‚0.7170982572544441 _runtime‚16876.096345857 0loss/FRt1/assignment_loss‚1.4377344085002697 0loss/FRt2/assignment_loss‚0.7551233517272132 3loss/SRqqt1/assignment_loss‚0.29445055816002313 0loss/FBt1/detection_loss‚0.38002688773684995 %loss/total_loss‚7.089504915000287 trainer/global_step‚19899¨Y¤ Ã< 4loss/FBt2/assignment_loss‚1.2500137249560338e-05 /loss/FRt1/detection_loss‚0.5189044265889576 1loss/SRqqt1/detection_loss‚0.7621442069042536 epoch‚49 % _timestamp‚1.7842896839709752e+09 3loss/FBt1/assignment_loss‚0.0007394015392905231 1loss/SRqqt2/detection_loss‚0.5931135278097645 /loss/FBt2/detection_loss‚0.7170982572544441 _runtime‚16876.096345857 0loss/FRt1/assignment_loss‚1.4377344085002697 0loss/FRt2/assignment_loss‚0.7551233517272132 3loss/SRqqt1/assignment_loss‚0.29445055816002313 0loss/FBt1/detection_loss‚0.38002688773684995 %loss/total_loss‚7.089504915000287 trainer/global_step‚19899 -loss/FRt2/detection_loss‚1.21111033478072 _step‚1640 2loss/SRqqt2/assignment_loss‚0.4190470538612506è Æú‘Ä<‚(:† ¡³èÒ½éÿ¢ gpu.0.gpu‚55.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ŠVÅ<‚(:“ ¡³èÒÉ÷¸¥ disk.sda.out‚ 706.5703125 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.9960357202204 disk.sda3.out‚ 706.5703125" disk.dm-0.out‚71904.0283203125 network.sent‚ 3.1701207e+07 network.recv‚ 4.449215e+06$ disk./.usageGB‚551.0747680664062* proc.memory.percent‚17.209172248840332 disk.dm-1.in‚ 0.0078125% disk.nvme0n1.out‚71904.0283203125& proc.memory.availableMB‚ 407.671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0.0625! disk.rbd1.out‚125.71923828125 disk.dm-1.out‚ 706.5703125 disk.rbd3.in‚0 disk.dm-0.in‚145023.25390625 proc.cpu.threads‚17 disk.sda.in‚ 0.0078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda3.in‚ 0.0078125$ memory_percent‚97.51176834106445# proc.memory.rssMB‚ 2819.55078125* disk./.usagePercent‚36.983090067464744 disk.rbd3.out‚0# disk.nvme0n1.in‚145023.25390625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚71904.0283203125Þó•žÆ<‚(:“ °³èÒ€ŸË¢ gpu.0.gpu‚87.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚Ü Ç<‚(:’ °³èÒ…•á¤ disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0.0625 disk.dm-0.in‚145023.25390625% disk.nvme0c0n1.in‚145023.25390625 disk.sda2.in‚0 disk.dm-0.out‚71912.55078125# disk.nvme0n1.in‚145023.25390625! disk.rbd1.out‚125.78173828125 disk.rbd0.in‚0 disk.dm-1.in‚ 0.0078125 disk.rbd3.out‚0# proc.memory.rssMB‚ 2820.20703125 proc.cpu.threads‚17 network.recv‚ 4.453284e+06 disk.drbd2165.out‚0 disk.sda.out‚ 706.97265625 disk.rbd3.in‚0 disk.sda2.out‚0 network.sent‚ 3.1764329e+07 disk.sda3.out‚ 706.97265625 disk.sda1.out‚0$ memory_percent‚97.44269847869873 cpu‚25.995790667732184) disk./.usagePercent‚36.98342978946268 disk.drbd2165.in‚0 disk.dm-1.out‚ 706.97265625 disk.sda1.in‚0$ disk./.usageGB‚551.0798301696777 disk.rbd0.out‚0% disk.nvme0c0n1.out‚71912.55078125 disk.sda.in‚ 0.0078125# disk.nvme0n1.out‚71912.55078125( proc.memory.availableMB‚ 418.98828125) proc.memory.percent‚17.21317768096924“Ê™4È</ _wandbruntime‚16915  _runtime‚16915zÉÈÉ< & lr-AdamW/pg1‚7.896269145405239e-09 & lr-AdamW/pg2‚7.896269145405239e-09 trainer/global_step‚19949 $ _timestamp‚1.784289723397312e+09 _runtime‚16915.52286375 _step‚1641eŸÍÊ<Ç trainer/global_step‚19949 $ _timestamp‚1.784289723397312e+09 _runtime‚16915.52286375 _step‚1641 & lr-AdamW/pg1‚7.896269145405239e-09 & lr-AdamW/pg2‚7.896269145405239e-09é gø4Ë</ _wandbruntime‚16915  _runtime‚16915\ø[!3&Ì<€P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21328126472192 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 _runtime‚16915.524083854 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21097023312896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3053140512661504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342953859 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3032043130970112 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77867897 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3032043130970112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42925516 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚19949 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184778963 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184768986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42925786 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3053140154283008 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21065176003070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3053140154283008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158174894 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21328031603712 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3016447668113408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342943880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34942108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚689135530874880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚667807385790464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3032043148009472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158174894 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21097023312896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184778963 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184768986 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21097364652032 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21097023312896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184778963 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184768986 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21097364652032 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42925516 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚19949 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184778963 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184768986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42925786 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3053140154283008 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21065176003070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3053140154283008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158174894 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21328031603712 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3016447668113408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342943880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ê cãȪ4Î</ _wandbruntime‚16916  _runtime‚16916‚›ÿ¿'Ï<‹N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158182526 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158182526 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184788029 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158182528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34943853 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842897240106578e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3032190371999744 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158182528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21329141195776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21329046334464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184778038 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42927765 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184778038 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21098035939840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3053288766325248 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3016594195824640 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3016594178785280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21098377286144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3032190389039104 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77871340 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3037660382881544 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21098035939840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42927487 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21066187056904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34943856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚689169860883456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3053288766325248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342970557 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚16916.136473907 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3053288407939584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3032190371999744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21065846506902}m'Ð<N FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21065846506902 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158182526 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158182526 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184788029 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158182528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34943853 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7842897240106578e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3032190371999744 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158182528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21329141195776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21329046334464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184778038 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42927765 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184778038 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21098035939840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3053288766325248 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3016594195824640 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3016594178785280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21098377286144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 155225600 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3032190389039104 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77871340 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3037660382881544 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21098035939840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42927487 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21066187056904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34943856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚689169860883456 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3053288766325248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342970557 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚16916.136473907 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3053288407939584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3032190371999744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888ë ±UC4Ñ</ _wandbruntime‚16916  _runtime‚16916ż‘ÆÒ<ˆ 0loss/FBt1/detection_loss‚0.30537107940154173 /loss/FRt2/assignment_loss‚0.689251971539676 &loss/total_loss‚6.8110801899037305 trainer/global_step‚19949 epoch‚49 3loss/FBt2/assignment_loss‚0.0004935356908793901 /loss/FRt1/detection_loss‚0.5119983888446247 0loss/SRqqt2/detection_loss‚0.607542373806886 _runtime‚16916.139069597 0loss/FRt1/assignment_loss‚1.5235189211611846 2loss/SRqqt2/assignment_loss‚0.4830533632378894 % _timestamp‚1.7842897240137346e+09 _step‚1644 .loss/FBt2/detection_loss‚0.455389573877815 2loss/SRqqt1/assignment_loss‚0.2683083798220612 3loss/FBt1/assignment_loss‚0.0024834898730506892 /loss/FRt2/detection_loss‚1.1472835626604903 1loss/SRqqt1/detection_loss‚0.8163855499876321¨æÃÓ< epoch‚49 3loss/FBt2/assignment_loss‚0.0004935356908793901 /loss/FRt1/detection_loss‚0.5119983888446247 0loss/SRqqt2/detection_loss‚0.607542373806886 _runtime‚16916.139069597 0loss/FRt1/assignment_loss‚1.5235189211611846 2loss/SRqqt2/assignment_loss‚0.4830533632378894 % _timestamp‚1.7842897240137346e+09 _step‚1644 .loss/FBt2/detection_loss‚0.455389573877815 2loss/SRqqt1/assignment_loss‚0.2683083798220612 3loss/FBt1/assignment_loss‚0.0024834898730506892 /loss/FRt2/detection_loss‚1.1472835626604903 1loss/SRqqt1/detection_loss‚0.8163855499876321 0loss/FBt1/detection_loss‚0.30537107940154173 /loss/FRt2/assignment_loss‚0.689251971539676 &loss/total_loss‚6.8110801899037305 trainer/global_step‚19949ì >Mt]3Ô<‚(:’ ¿³èÒ鳤 gpu.0.gpu‚88.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚'B0æj1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C©5 Õ<‚(:• ¿³èÒå鶥 disk.rbd1.in‚0.0625 disk.dm-1.in‚ 0.0078125 network.sent‚ 3.1768445e+07 disk.rbd0.out‚0% disk.nvme0c0n1.in‚145023.25390625( disk.nvme0c0n1.out‚71924.35400390625 disk.sda3.out‚ 707.2578125 network.recv‚ 4.45616e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 0.0078125$ memory_percent‚97.44107723236084 disk.dm-0.in‚145023.25390625# disk.dm-0.out‚71924.35400390625 disk.sda.out‚ 707.2578125( disk./.usagePercent‚36.9837656713476 disk.sda1.in‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.out‚ 707.2578125 cpu‚25.99555648551547& disk.nvme0n1.out‚71924.35400390625 disk.drbd2165.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2820.20703125$ disk./.usageGB‚551.0848350524902 disk.sda2.in‚0( proc.memory.availableMB‚ 419.25390625 proc.cpu.threads‚17 disk.sda2.out‚0! disk.rbd1.out‚125.78173828125) proc.memory.percent‚17.21317768096924# disk.nvme0n1.in‚145023.25390625 disk.sda1.out‚0~¬š‹Ö<‚(:’ γèÒÕ™¸¢ gpu.0.gpu‚74.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚269.087' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.81739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uµæ×<‚(:” γèÒ¨ÂÙ¤# proc.memory.rssMB‚ 2820.20703125 network.recv‚ 4.460522e+06 disk.drbd2165.in‚0 disk.sda.out‚ 707.51953125 disk.sda3.in‚ 0.0078125! disk.dm-0.out‚71933.107421875 proc.cpu.threads‚17 disk.sda.in‚ 0.0078125# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0 disk.dm-1.out‚ 707.51953125! disk.rbd1.out‚125.84423828125$ disk./.usageGB‚551.0921287536621$ disk.nvme0n1.out‚71933.107421875 disk.rbd0.out‚0% disk.nvme0c0n1.in‚145023.25390625& disk.nvme0c0n1.out‚71933.107421875 disk.sda2.out‚0 disk.rbd1.in‚0.0625) proc.memory.percent‚17.21317768096924 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚145023.25390625 disk.sda1.out‚0( proc.memory.availableMB‚ 419.03515625 cpu‚25.99531244377016 network.sent‚ 3.1831514e+07 disk.sda2.in‚0) disk./.usagePercent‚36.98425515775307 disk.rbd0.in‚0 disk.sda3.out‚ 707.51953125 disk.dm-1.in‚ 0.0078125 disk.rbd3.in‚0$ memory_percent‚97.44241237640381(!€[Ø<‚(:’ ݳèÒƨ֢ gpu.0.gpu‚67.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ^ÃÙ<‚(:” ݳèÒÙª—¥$ disk./.usageGB‚551.1038780212402$ disk.nvme0n1.out‚71946.193359375 disk.sda.in‚ 0.0078125 disk.sda2.in‚0# proc.memory.rssMB‚ 2820.20703125) proc.memory.percent‚17.21317768096924 proc.cpu.threads‚17 network.recv‚ 4.464219e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 707.85546875 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚71946.193359375 network.sent‚ 3.1836561e+07) disk./.usagePercent‚36.98504366095853 disk.rbd0.in‚0% disk.nvme0c0n1.in‚145023.25390625 disk.sda3.out‚ 707.85546875# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0& disk.nvme0c0n1.out‚71946.193359375 disk.sda2.out‚0( proc.memory.availableMB‚ 418.98046875 disk.sda1.out‚0 disk.rbd1.in‚0.0625$ memory_percent‚97.44274616241455 cpu‚25.99506496738629 disk.sda.out‚ 707.85546875! disk.rbd1.out‚125.84423828125 disk.dm-0.in‚145023.25390625 ¥ÿû4Ú</ _wandbruntime‚16955  _runtime‚16955ô)ÞžÌÛ<Æ _step‚1645 ' lr-AdamW/pg1‚3.0358758448500848e-12 ' lr-AdamW/pg2‚3.0358758448500848e-12 trainer/global_step‚19999 % _timestamp‚1.7842897630992596e+09 _runtime‚16955.226389865ÍôYZÑÜ<Ë _step‚1645 ' lr-AdamW/pg1‚3.0358758448500848e-12 ' lr-AdamW/pg2‚3.0358758448500848e-12 trainer/global_step‚19999 % _timestamp‚1.7842897630992596e+09 _runtime‚16955.226389865í 1uÔà4Ý</ _wandbruntime‚16955  _runtime‚16955ó¬ˆ](Þ<P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185223438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21115451940088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21379196874752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚690857936208896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1646 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43027642 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669478842329088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43027910 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185233415 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3044888653647096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21148060153856 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3044889011236346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842897631000574e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3039405182451712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21147718814720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 trainer/global_step‚19999 _runtime‚16955.226477901 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185223438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669478823716864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21115792489978 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158556494 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3060552901266432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚690858039203840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158556496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158556496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21379112492032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158556494 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3039405199491072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3023773201707008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35029361 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3060553259644928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3023773218746368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35029358 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343779932 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78057271 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21148060153856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21147718814720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 343789911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3039405182451712 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185233415 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0…nðÛ (ß<†P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185223438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669478823716864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21115792489978 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158556494 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3060552901266432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚690858039203840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158556496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158556496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21379112492032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 155225600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158556494 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3039405199491072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3023773201707008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35029361 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3060553259644928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3023773218746368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35029358 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343779932 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78057271 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21148060153856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21147718814720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 343789911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3039405182451712 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185233415 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185223438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21115451940088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21379196874752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚690857936208896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1646 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43027642 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669478842329088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43027910 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185233415 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3044888653647096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21148060153856 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3044889011236346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842897631000574e+09 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3039405182451712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21147718814720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 trainer/global_step‚19999 _runtime‚16955.226477901 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2î ¼Å­m4à</ _wandbruntime‚16955  _runtime‚16955>ëÛPá<ŠN =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3060701513308672 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78061052 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185232490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3060701513308672 trainer/global_step‚19999 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3023919712378880 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21116803543812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3045036532962052 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78060782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669512252475392 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3039552423481344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35031106 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43029679 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158564126 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3039552440520704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚690892489657344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16955.837039798 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343806609 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3023919729418240ÐÑ*'â<N GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3060701513308672 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78061052 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185232490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3060701513308672 trainer/global_step‚19999 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3023919712378880 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21116803543812 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3045036532962052 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78060782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669512252475392 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3039552423481344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35031106 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43029679 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158564126 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3039552440520704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚690892489657344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16955.837039798 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343806609 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3023919729418240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ï Hןè4ã</ _wandbruntime‚16955  _runtime‚16955&Á‰ ä<ƒ 2loss/SRqqt1/assignment_loss‚0.3861391980885658 /loss/FRt1/detection_loss‚0.5064563454091348 /loss/FRt2/detection_loss‚1.2630266601919073 _runtime‚16955.839167495 _step‚1648 2loss/FBt1/assignment_loss‚0.003164084441766262 1loss/SRqqt1/detection_loss‚0.8557993994094432 # _timestamp‚1.78428976371383e+09 0loss/FBt2/detection_loss‚0.36931152749704776 %loss/total_loss‚6.918483475688907 .loss/FRt2/assignment_loss‚0.75539390290225 2loss/SRqqt2/assignment_loss‚0.4055978226794332 1loss/SRqqt2/detection_loss‚0.5278243658919151 /loss/FBt1/detection_loss‚0.3355619239238597 trainer/global_step‚19999 1loss/FBt2/assignment_loss‚0.01882936579980594 epoch‚49 0loss/FRt1/assignment_loss‚1.4913788794537786•¶§?å<ˆ .loss/FRt2/assignment_loss‚0.75539390290225 2loss/SRqqt2/assignment_loss‚0.4055978226794332 1loss/SRqqt2/detection_loss‚0.5278243658919151 /loss/FBt1/detection_loss‚0.3355619239238597 trainer/global_step‚19999 1loss/FBt2/assignment_loss‚0.01882936579980594 epoch‚49 0loss/FRt1/assignment_loss‚1.4913788794537786 2loss/SRqqt1/assignment_loss‚0.3861391980885658 /loss/FRt1/detection_loss‚0.5064563454091348 /loss/FRt2/detection_loss‚1.2630266601919073 _runtime‚16955.839167495 _step‚1648 2loss/FBt1/assignment_loss‚0.003164084441766262 1loss/SRqqt1/detection_loss‚0.8557993994094432 # _timestamp‚1.78428976371383e+09 0loss/FBt2/detection_loss‚0.36931152749704776 %loss/total_loss‚6.918483475688907ð ~û™î‘æ<‚(:† ì³èÒÆÞÁ¤ gpu.0.gpu‚58.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 *¹sç<‚(:ˆ ì³èÒºâã¦# proc.memory.rssMB‚ 2820.20703125 network.sent‚ 3.184165e+07 network.recv‚ 4.467275e+06) disk./.usagePercent‚36.98547170555578 disk.rbd1.in‚0.0625 disk.sda1.out‚0 proc.memory.availableMB‚419 cpu‚25.99687195795895 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125 disk.sda3.out‚ 708.109375 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 708.109375 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 708.109375 disk.sda2.in‚0# disk.nvme0n1.in‚145023.25390625 disk.sda1.in‚0" memory_percent‚97.442626953125 proc.cpu.threads‚17$ disk./.usageGB‚551.1102561950684% disk.nvme0c0n1.in‚145023.25390625( disk.nvme0c0n1.out‚71955.53955078125! disk.rbd1.out‚125.90673828125 disk.rbd3.in‚0& disk.nvme0n1.out‚71955.53955078125) proc.memory.percent‚17.21317768096924 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚145023.25390625# disk.dm-0.out‚71955.53955078125M׉4è</ _wandbruntime‚16973  _runtime‚16973½Ô€ =é<‡{ (particle/accuracy‚0.8415062427520752 +particle/specificity‚0.9036658406257629 'particle/f_score‚0.6246719360351562 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27394968271255493 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5936349034309387 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_2_of_2‚1 6validation_average_jet_accuracy‚0.3713553845882416 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897313714027405 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5952374339103699 particle/accuracy_2_of_4‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2160274237394333 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2378336787223816 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22576673328876495 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6555813550949097 +jet/accuracy_3_of_3‚0.05731193348765373 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,jet/accuracy_4_of_4‚0.011098379269242287 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2954205274581909 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6250426173210144 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24297332763671875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762619495391846 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8021252155303955 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953824281692505 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚19999 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.658693790435791 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3549778461456299 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7759550213813782 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6136214733123779 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7"Purity/*FRt1SRqqt1FBt/event_purity‚0.37242391705513 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8971921801567078 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19793203473091125 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043658751994371414 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_2‚0.7877848744392395 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7278076410293579 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022459974512457848 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 *jet/accuracy_1_of_3‚0.9248284697532654 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9156632423400879 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6925165057182312 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_4‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2501610219478607 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762619495391846 particle/accuracy_2_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947301149368286 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5456558465957642 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021032636985182762 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25073766708374023 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5738587975502014 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6665154695510864 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2722296118736267 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20418885350227356 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6219837069511414 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6305881142616272 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6543384194374084 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5867990255355835 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2722240090370178 +particle/sensitivity‚0.6142222881317139 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8547785878181458 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6573099493980408 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7185044884681702 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0893634632229805 +jet/accuracy_2_of_2‚0.15793344378471375 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037037014961243 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21090693771839142 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2622140944004059 particle/accuracy_1_of_2‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7218806743621826 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5924755930900574 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 _step‚1649 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6420996189117432 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5836424827575684 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7278532981872559 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047128789126873016 *jet/accuracy_2_of_4‚0.8018405437469482 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23845195770263672 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037037014961243 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6097307205200195 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6195706129074097 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03585115447640419 particle/accuracy_5_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23761804401874542 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7686331868171692 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875089526176453 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2724328637123108 particle/accuracy_3_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7842897809376748e+09 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7267573475837708 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7612698674201965 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.041441187262535095 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6588912606239319 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22437813878059387 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5301562547683716 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_2_of_3‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.263298362493515 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9927353858947754 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5982125401496887 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.764668345451355 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7334467172622681 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5725581645965576 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6234893798828125 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6891278624534607 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28709539771080017 *jet/accuracy_1_of_4‚0.9857255220413208 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02115723118185997 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2640146017074585 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21924608945846558 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6246398091316223 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 .particle/accuracy_4_of_4‚0.948646068572998 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6550284624099731 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4511604905128479 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5133333206176758 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.530368447303772 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5723271369934082 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5996617674827576 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022613270208239555 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32947641611099243 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3198842406272888 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20643974840641022 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9906184673309326 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35528329014778137 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5996617674827576 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4885523319244385 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07827772200107574 _runtime‚16973.064176655 )loss/val_total_loss‚8.091816522390943 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.254358172416687 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7296673655509949 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6310185790061951 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7958320379257202 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23920990526676178 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 epoch‚49 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2488555610179901 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8457856774330139 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2622140944004059 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6229508519172668 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +jet/accuracy_1_of_1‚0.32769596576690674 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5424817204475403 +jet/accuracy_3_of_4‚0.07847467809915543 particle/accuracy_3_of_3‚1eYy5ê<Œ{ :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03585115447640419 particle/accuracy_5_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23761804401874542 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7686331868171692 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875089526176453 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2724328637123108 particle/accuracy_3_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7842897809376748e+09 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7267573475837708 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7612698674201965 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.041441187262535095 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6588912606239319 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22437813878059387 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5301562547683716 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_2_of_3‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.263298362493515 7 Purity/*FRt1SRqqt1b}]8FBt/FBt_purity‚0.9927353858947754 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5982125401496887 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.764668345451355 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7334467172622681 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5725581645965576 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6234893798828125 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6891278624534607 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.28709539771080017 *jet/accuracy_1_of_4‚0.9857255220413208 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02115723118185997 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2640146017074585 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21924608945846558 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6246398091316223 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 .particle/accuracy_4_of_4‚0.948646068572998 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6550284624099731 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4511604905128479 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5133333206176758 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961491823196411 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.530368447303772 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5723271369934082 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5996617674827576 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022613270208239555 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.016731999814510345 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32947641611099243 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3198842406272888 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964785575866699 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20643974840641022 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9906184673309326 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35528329014778137 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5996617674827576 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4885523319244385 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07827772200107574 _runtime‚16973.064176655 )loss/val_total_loss‚8.091816522390943 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.254358172416687 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7296673655509949 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6310185790061951 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7958320379257202 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23920990526676178 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 epoch‚49 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2488555610179901 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8457856774330139 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2622140944004059 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6229508519172668 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9900238513946533 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7278532981872559 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +jet/accuracy_1_of_1‚0.32769596576690674 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6015625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968443512916565 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5424817204475403 +jet/accuracy_3_of_4‚0.07847467809915543 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8415062427520752 +particle/specificity‚0.9036658406257629 'particle/f_score‚0.6246719360351562 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27394968271255493 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5936349034309387 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_2_of_2‚1 6validation_average_jet_accuracy‚0.3713553845882416 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9897313714027405 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5952374339103699 particle/accuracy_2_of_4‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2160274237394333 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2378336787223816 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22576673328876495 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6555813550949097 +jet/accuracy_3_of_3‚0.05731193348765373 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972948431968689 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,jet/accuracy_4_of_4‚0.011098379269242287 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2954205274581909 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6250426173210144 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24297332763671875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8762619495391846 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.8021252155303955 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953824281692505 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚19999 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.658693790435791 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3549778461456299 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7759550213813782 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6136214733123779 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7"Purity/*FRt1SRqqt1FBt/event_purity‚0.37242391705513 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8971921801567078 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19793203473091125 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043658751994371414 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_2‚0.7877848744392395 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7278076410293579 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.022459974512457848 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.016731999814510345 *jet/accuracy_1_of_3‚0.9248284697532654 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9156632423400879 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6925165057182312 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_4‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2501610219478607 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8762619495391846 particle/accuracy_2_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947301149368286 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5456558465957642 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.021032636985182762 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25073766708374023 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5738587975502014 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6665154695510864 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2722296118736267 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20418885350227356 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09416667371988297 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6219837069511414 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6305881142616272 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6543384194374084 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5867990255355835 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2722240090370178 +particle/sensitivity‚0.6142222881317139 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8547785878181458 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6573099493980408 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7185044884681702 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0893634632229805 +jet/accuracy_2_of_2‚0.15793344378471375 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7037037014961243 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21090693771839142 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2622140944004059 particle/accuracy_1_of_2‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7218806743621826 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5924755930900574 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 _step‚1649 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6420996189117432 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5836424827575684 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7278532981872559 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9971702098846436 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.047128789126873016 *jet/accuracy_2_of_4‚0.8018405437469482 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23845195770263672 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909517168998718 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7037037014961243 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6097307205200195 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6195706129074097 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685ñ ÁÄ*“Rë< g2vc1w2gj@ ô³èÒУ•Ï0`Trainer.fit` stopped: `max_epochs=50` reached. ",4¡ì< g2vc1w2gjŽ õ³èÒ€êÓ˜|Epoch 49/49 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 400/400 0:05:19 • 0:00:00 1.26it/s v_num: 1w2gâFý.%í< g2vc1w2gj õ³èÒø€é˜ ÊÎœG4î</ _wandbruntime‚16975  _runtime‚16975F±.Qï<‚(’Ï„